• Announcements

    • Right as Reign

      Logging in   05/10/2017

      If you had an account and can't log in as of 5/9/17, this may be because of a change in logins with new forum software. You can log in using your publicly displayed name (not your username) or your email address and the password you used before.   If you have problems with this, please ask any of the mods or admins. 
Sign in to follow this  
Followers 0
The Almighty Doer of Stuff

Housekeeping: Revised Blades of Avernum Ratings System

386 posts in this topic

Quote:
No location will work for absolutely everyone. (And it's not like there aren't indirect methods if someone has something they really want to say.)

Dikiyoba.


But that's something that should be avoided, not counted on. And true, one could just Email the review to the designer himself, but then what use is it to the player and/or anyone else

Quote:
I still think we need some description of what a 1, 2, 3 or D, C, C+ is. Teachers don't mark without a list of letter grade descriptions, or one is available at the very least. I realise that everyone has a different idea of what outstanding or first class combat or plot is. Yet I still feel we need at least a sketch of what each letter or number actually means. We should be trying to standardise the reviewing process. Every reviewer should look at the list of number descriptions and then decide what number that scenario deserves. Otherwise it is like pulling a number out of the air. "I like 5s today, maybe a 5 will be a 7 tomorrow".


Second. Like I mentioned earlier, this is the state of things. If not for the direct benefit of the Designer, the Player would benefit at least from a consistent and sensible rubric.

Share this post


Link to post
Share on other sites

I disagree. As long as reviewers try to be internally consistent, it should be fine. Wisdom of the crowds, law of large numbers, etc.

 

We could also provide median scores in addition to mean scores.

 

EDIT: And yes, specifying a 1-10 scale or 0-10 scale should be done. But aside from saying that a reviewer should "like" a scenario they rate 7 more than they "like" a scenario they rate 6.5, we don't need much. ("Like" or "think is better than" both work)

 

Although we should discourage vindictive/adjusting ratings that are exaggerated to shift the mean.

 

EDIT 2: Why couldn't we host CSR on these boards? A forum right next to the 3 Blades forums.

Share this post


Link to post
Share on other sites

I agree with Drakefyre. I don't see the need to establish a rubric or any other concrete method of scoring. Having recently read some reviews, I noticed that all of the reviews offered useful points to the designer, yet there was a variety of methods used to score the scenario, and all of the methods seemed to produce a reasonable conclusion. Plus, some people will likely be deterred by having to conform to a set guideline.

Share this post


Link to post
Share on other sites

I agree with Drakefyre, and think that just hosting the reviews here in another forum would minimize all of this fuss. Subforum of BoA and all that. I'd be happy to moderate that.

 

Also, that table is intimidating, but I suppose it works.

Share this post


Link to post
Share on other sites

I'd be fine with moving the CSR here, but I always assumed that Jeff objected (because spiderweb had that terrible review section on its terrible scenario tables.)

 

If this isn't the case, or if Jeff's opinion is different now that they don't even pretend to keep up the tables, then cool. I don't know why it didn't come up when we were moving the CSR to SV in the first place....

 

Edit: Also it could be a subforum here to avoid cluttering the main page.

Share this post


Link to post
Share on other sites

Here? I honestly don't believe that Jeff will allow the CSR on his own website. He doesn't really support the Blades Community as it is, so what makes you guys think he'll put it up? But, hey, it makes everyone happy, so its definitely worth a shot. At worst, he'll just say no, and we would be no further back then before.

 

(also, we should probably ask him relatively soon, because he's supposed to start Beta testing A6 in a few weeks, and will be more likely to just say no out of annoyance then anything else.)

Share this post


Link to post
Share on other sites

The Blades Forge seems like a pretty neutral location to me, really. Does anyone have some strong dislike for Arancaytar or Ermarian.net? Plus, unlike a message board-based system like the current CSR, a Blades Forge-based system has more flexibility and searchability.

 

I was talking to Bain-Ihrno last night, and we thought of some ideas. Perhaps, in addition to the "poll thread"-style rating/review/discussion system, we could have a button next to the results that says "Ratings By User" that would display each individual rating and the person who rated it that way. Perhaps you could even choose to eliminate the ratings of certain people who you tend to disagree with, or search only for ratings by people you tend to agree with. This is better than the thumbs up/down system that was proposed once, because it's more personalized to the user and doesn't make other reviewers who might get too many thumbs down unjustly feel like they're being bullied.

 

As for Nioca's rubric, I find myself disagreeing with many aspects of it. This is one of the problems with the huge rubric that Drakey mentioned: not everyone agrees with what makes an aspect of a scenario good and what doesn't. In addition, that's a LOT of reading to expect everyone who tries to use the system to read before they can rate scenarios.

 

While I don't think we should make the system overly simple to the point where it's not functional or practical, I strongly believe we should make it as easy as possible to use the system. Complex rubrics, while useful in theory, will tend to just discourage people from using the system. Furthermore, I could see the problem arising of people starting a fuss about specific people applying the rubric unfairly. With no rubric, it's all personal opinion and everyone knows that.

 

Besides, rubricless rating systems still work well in other applications. For instance, on Amazon.com, you rate a product from 1 to 5 stars. No rubric is provided, but the reviews still serve a useful purpose. You select a number of stars, and read the reviews that gave that number of stars, and you figure out which reviews tend to scrutinize the item most fairly and logically. You may find that all the 1-star reviews are by people who didn't read the instructions, and thus you can ignore them. Alternately, you may find that the 5 star reviews are from people who don't know how to tell whether the product in question serves its purpose well compared to similar products, and you can ignore them.

 

I think the system Bain and I came up with for ignoring users you frequently disagree with or searching only for ratings by people you tend to agree with fits its purpose well in that regard. Even people who don't review or discuss don't pose a problem, because you can base whether you agree with that person on whether you agree with reviewers who rated the scenario similarly.

Share this post


Link to post
Share on other sites

So, back to the questions.

 

1 - Do we need to change how scenarios are rated? You know, the methodology.

2 - Does the CSR need moving? Certain answers to #1 would mandate it, others wouldn't.

 

And no, moving it because some folks can't post on SV is not a valid answer. Some folks can't post here. Actually, probably more BoX people are restricted from posting here than there.

Share this post


Link to post
Share on other sites

1. I have heard some ideas that I think would be vastly better. I say we pick the best one, and change it.

2. Yes. No one should have to wade through porn to read a scenario rating.

Share this post


Link to post
Share on other sites

I'm going to agree with Salmon. If we're going to move it, it should be either because we're putting in some new rating system that can't be done using the current forum setup, or it should be moved here where it would be most widely accessible (And I'm not talking about forum preferences of a handful of users here, I'm talking about being accessible to all the users who don't frequent any satellite, and could care less about SV vs CRF.) Moving the CSR from one satellite to another satellite with a stricter CoC is just a waste of time. We keep our shenanigans out of the holy grounds of the CSR, so you can just drop the stupid "Think of the children!" routine.

 

As far as new rating systems go, there are a few major problems. Firstly, the work that's going to go into changing it. Since the new setups require fancy web scripting and such, this basically would all fall on the shoulders of one or two users (or just Aran.) Secondly, we have hundreds of old reviews that won't fit this new format, what do we do with those? Do we ask users to repost all their old scores in the new format? That would be a huge pain, and totally pointless. Some reviewers are no longer active members, and others are still active but don't spend enough time here anymore to be expected to move all their reviews. We could list the old reviews alongside the new reviews, just the way they are as a block of text. But that just comes off as half-assed to have a huge portion of the CSR be in one format and a few of the new ratings be in a different one. So do we throw all the old reviews away and start fresh? That's a pretty big waste since expecting a new CSR to catch up in size to the current one is probably not reasonable.

 

So it's not just a matter of "This looks like a better system, and we're jonesing for change, so let's do it!"

Share this post


Link to post
Share on other sites
Originally Posted By: Smiley-Faced Duck
2. Yes. No one should have to wade through porn to read a scenario rating.

How is that remotely relevant to where the CSR is now?

Share this post


Link to post
Share on other sites

CSR on SV never made sense to me in the first place.

 

The final number for a CSR rating seems like crap to me. How do you rate a scenario with a single number? I would've played a hell of a lot more BoE scenarios if I could've ever found what I was looking for. My favorite scenario, The Election, was buried down in the 7-ish area, and I don't have any idea how I ever found it in there. Scenarios that I found deeply frustrating were all the way up in the 9's. CSR's numbers concealed more than they revealed, as far as I was concerned.

 

How do you compare Nobody's Heroes, which is a short, humor-based scenario with basically no combat, to Exodus, which is a very long, serious scenario with tons and tons of combat, if you're using just a single number? How do you compare Nine Variations on Point B, which is a meta-scenario port, to the High Level Party Maker, which is a utility scenario, with just a single number? The number is useless — worse than useless, because it directs attention away from the substantive issues in the scenario.

 

What we need is a set of tags (number of towns, number of outdoor sections, etc.), some of which are subjective (degree of polish), ideally in a searchable format. If I want to pull up a short, humor-based scenario that is well-polished, with or without combat, I should be able to do that quickly and easily. Right now, I can't.

 

Feedback for designers can take some other completely different form. But this is what the players need.

Share this post


Link to post
Share on other sites

Kel has a legitimate beef here, imo, and is really talking about the only reason for making any change. After all, the purpose of this whole discussion is to increase the audience of Blades.

 

Now, most of us who review already break it out into various subcomponents, giving our x-10 rating for each, and then combining and using some sort of averaging (see Nioca's in SV for reference, no link provided). If there was a way of making that a standard, then that would be one system. Instead of one number, there could be 4 (plot, combat, etc etc).

 

Another would be to break out the existing single score ranking system into multiple grouplings, based on those tags. A scenario might qualify under two or more groups as well. So, a Combat group, Large group, Small group, Plot group. The designer could specify, or in the case of older BoE, a resident B(ain)oEophile could do it for us.

Share this post


Link to post
Share on other sites
Originally Posted By: Naughty Salmon
So, back to the questions.

1 - Do we need to change how scenarios are rated? You know, the methodology.
2 - Does the CSR need moving? Certain answers to #1 would mandate it, others wouldn't.

And no, moving it because some folks can't post on SV is not a valid answer. Some folks can't post here. Actually, probably more BoX people are restricted from posting here than there.


Exactly. There are some people who can't post everywhere, for whatever reason. So why not move it to a place where differnces don't matter?

Share this post


Link to post
Share on other sites

Salmon, when you say "Some folks can't post here" I assume you're referring to TM and TGM. Other members of the community are more than happy to channel them for legitimate business and the administration seems okay with that, so I don't see that as too much of a problem. SW is probably better than SV if we keep it message-board based.

 

Anyway, I've modified my Blades Forge-based idea yet again in light of recent discussion, so here it is:

 

---

 

There would be three specific Quality rating categories on a scale of 1-10, with 1 being absolutely bereft of any sort of redeeming quality whatsoever and 10 being absolutely perfect, all listed as numbers only except 1 and 10. For each of these ratings there would also be "Not Applicable" and "Abstain/No Opinion" options. There would also be a similarly-scaled Overall Quality rating, except it would be to one decimal place, like CSR.

 

There would also be three Genre scales. These would have five (or whatever odd number) descriptive ratings with an clearly defined center, that serve to indicate what sort of scenario it is.

 

All ratings would be optional except (possibly) Overall Quality, which might be required for all rating submissions. A user may modify his past ratings at will.

 

The above describe Rating system will be placed at the top of a scenario's Discussion Thread for reviews and other discussion, sort of like a poll on a typical message board (only better). Results would have to be visible by default, unlike message board polls, because first you look at the rating to decide what to play, then you play it, and only then do you actually rate it. Next to the results will be a button for "Ratings By User" that will display the ratings by user and allow you to uncheck or recheck boxes next to each user, in order to exclude or include a specific user's ratings in the calculated, searchable averages.

 

Below the Ratings would be a Discussion Thread for textual reviews and discussions. Spoiler tags MUST be enforced, but spoilers are allowed. You can discuss the player-end aspects of the scenario, the designer-end aspects, or both; it doesn't matter. The actual rating aspect of the system is strictly player-oriented, however. This is both so that players won't have to deal with scenarios being rated highly for things that don't affect play, and to prevent people who don't know what they're talking about from bringing down advanced scenarios. (Actually, perhaps there could be an additional scale that's only available if you've released a scenario. That can be discussed later, though.)

 

Lastly, as is already present on the Blades Forge, you can Tag the scenario with keywords that indicate noteworthy aspects of the scenario.

 

---

 

The [b[Overall Quality Rating[/b] is independent, but has an optional "Calculate Average" button.

 

Each of the three specific categories would also have an explanation:

 

1. Writing

This category encompasses storyline, plot, themes, dialogue, descriptions, cliché or lack thereof, etc. The literary aspects of the scenario.

 

2. Gameplay

This category encompasses combat, puzzles, balance (how well the combat, treasure, EXP, etc. interact), engine modifications (such as the water system in "Destiny of the Spheres" or the rune system in "Nebulous Times Hence"), flow (or something, not sure what to call it. For instance, do you constantly have to blindly guess what you have to do or say to progress the scenario, or are the outdoors so unnecessarily large that it takes three weeks real time to get from one town to the next?), etc. What the player actually does while playing the scenario.

 

3. Aesthetics, Functionality, Etc.

This category includes graphics, spelling and grammar, town design, cutscenes, technical noding/scripting, bugginess, whether the scenario contains far too much gratuitous foul language/lewdness/etc. (if that's important to the person rating the scenario), artistic merit, etc. The presentation of the scenario, things whose primary purpose is to enhance (or detract from) immersion in the scenario. I know some people want a separate scale for graphics so we can exalt Ephesos's pretty trees, but ultimately I feel that people who are playing these games generally aren't doing it for the graphics, so I don't feel it's that important. If you want, you can always Tag the scenario with keywords like "beautiful" or "visually stunning" or whatever.

 

---

 

After those categories, the Genre Scales:

 

1. Serious<->Humorous +N/A (in case someone releases a scenario with nothing but combat and no text, which for all we know may happen, we'll need an N/A checkbox)

 

2. Hack-and-Slash Combat and No Puzzles<->Strategic Combat and/or Puzzle-Heavy +No Combat or Puzzles (I think strategic combat and puzzles should be together because really, strategic combat is a sort of puzzle. This scale will move right if there are puzzles, even if there is hack'n'slash or no combat, for instance, or if there's tactical combat but not puzzles, or both.)

 

3. Linear<->Open-Ended (Can anyone think of a scenario where this might need an N/A? Not necessarily a BoX "scenario" but scenario in the literal sense.)

 

---

 

I don't think we need a rating scale for short/long/epic, because the length of the scenario isn't subject to opinion. We could just include a Number of Towns and Size of Outdoors field in the database, I think. Same goes for recommended party level/prefab/singleton. We should be careful not to just use the level listed in the scenario, though, if the readme says otherwise. I don't know if this matters much in BoA, but in BoE, some scenarios were listed as Very High level just to prevent monster HP doubling. Lastly, for utility scenarios, the "genre" really isn't open to interpretation either. The fact that it is a utility scenario and perhaps a more specific categorical tag should be included in the database when the scenario is posted.

 

---

 

Of note is that the Overall Quality rating is the same scale as CSR's system, which makes it easy for CSR ratings to be imported into the Blades Forge (or whereever) system. The ratings would go into Overall Quality, and the reviews would go into the discussion thread. As with the original Blades Forge policy regarding old scenarios whose authors are no longer around, old CSR reviews will be placed under the control of the user who posted them when they register at the Blades Forge. This will of course be done manually, although the original porting from CSR to Blades Forge could probably be automated.

 

---

 

Any comments are welcome.

Share this post


Link to post
Share on other sites
Originally Posted By: The Almighty Doer of Stuff
Besides, rubricless rating systems still work well in other applications. For instance, on Amazon.com, you rate a product from 1 to 5 stars. No rubric is provided, but the reviews still serve a useful purpose. You select a number of stars, and read the reviews that gave that number of stars, and you figure out which reviews tend to scrutinize the item most fairly and logically. You may find that all the 1-star reviews are by people who didn't read the instructions, and thus you can ignore them. Alternately, you may find that the 5 star reviews are from people who don't know how to tell whether the product in question serves its purpose well compared to similar products, and you can ignore them.
The Blades Forge already has a rating system just like this, though I'm not entirely sure if it's available on the scenario pages.

Originally Posted By: Naughty Salmon
Another would be to break out the existing single score ranking system into multiple grouplings, based on those tags. A scenario might qualify under two or more groups as well. So, a Combat group, Large group, Small group, Plot group. The designer could specify, or in the case of older BoE, a resident B(ain)oEophile could do it for us.
Again, this is possible on the Blades Forge by using labels. There's no limit to what kind of labels can be applied to a scenario.



Of course, with some programming ADoS's idea above could also be implemented at the Blades Forge. The discussion thread already exists in the form of comments, doesn't it? Or if not it shouldn't be too hard to add since it's implemented elsewhere. Alternatively it could be placed in the Blades Forge forum with a link to the scenario page.

Share this post


Link to post
Share on other sites

This is also readily available (responding to Crazy Minstrel) at the SV within the current tables. Just break the master list into multiple lists. This can be done manually by anyone with mod access to that forum.

Share this post


Link to post
Share on other sites
Quote:
I just checked, and couldn't find any porn in the current CSR. Could you PM me the link please?

While there is nothing of that nature in either of the CSR forums I really just hate the idea of even having to go to that site when I know it really is in the other forums.

Share this post


Link to post
Share on other sites
Originally Posted By: Drakefyre
I disagree. As long as reviewers try to be internally consistent, it should be fine. Wisdom of the crowds, law of large numbers, etc.


Do the majority of BOX scenarios have a sufficient number of reviews for that to work?

Share this post


Link to post
Share on other sites

No, and since initial reviews influence subsequent reviews, it doesn't end up working out that way even when there are enough.

Share this post


Link to post
Share on other sites
Originally Posted By: Smiley-Faced Duck
Quote:
I just checked, and couldn't find any porn in the current CSR. Could you PM me the link please?

While there is nothing of that nature in either of the CSR forums I really just hate the idea of even having to go to that site when I know it really is in the other forums.


So, should we take it off the porn-laden internets altogether? I'm not saying there should (or shouldn't) be porn on any site, but I know it isn't in the CSR, or any of the BoX fora in Shadowvale. Here is the direct link to the BoA forums, and here is a similar pointer for the BoE forums. No need to ever chance a misclick onto the various topics in other places on the internet which may or may not (not that there is anything wrong with it) contain porn.

Share this post


Link to post
Share on other sites

...keep it civil, folks.

 

There's a whole lot of satellite-board squabbling going on right now. Some of it is legitimate, some of it isn't. I don't care about it right now, because this isn't about your personal drama. It's about Blades. Act like it for once, k?

Share this post


Link to post
Share on other sites

If all the discussion was rooted in said personal drama, then it should stop.

Share this post


Link to post
Share on other sites

Salmon, I believe you are not understanding, or not trying to understand, that porn on the website with the CSR *period* is still a legitimate problem with some people, including me.

 

Excellent post ADoS. A few little things

 

Quote:
This category includes graphics, spelling and grammar

Wouldn't Spelling and Grammar be in the Writing category? I mean, its got more to do with plot and atmosphere and such then it has to do with "Aesthetics".

 

Quote:
I don't think we need a rating scale for short/long/epic, because the length of the scenario isn't subject to opinion. We could just include a Number of Towns and Size of Outdoors field in the database, I think

Sound suggestion.

 

Quote:
3. Linear<->Open-Ended (Can anyone think of a scenario where this might need an N/A? Not necessarily a BoX "scenario" but scenario in the literal sense.)

I'm not sure. IMAGINE THIS SCENARIO maybe? It's not really meant to be a "real" scenario anyway.

 

Quote:
Do the majority of BOX scenarios have a sufficient number of reviews for that to work?

Unfortunately not. Some scores for BoE CSR just give numbers, for instance. frown If this takes off, I suppose we'll have to try something, though. That might include a few reviews that just give scores we might have to knock off, and for BoE scenario's like the Little Girl we could just put all the numbers to zero, or as close as can be gauged by whoever is putting them up.

Share this post


Link to post
Share on other sites
Originally Posted By: Master Ackrovan
Salmon, I believe you are not understanding, or not trying to understand, that porn on the website with the CSR *period* is still a legitimate problem with some people, including me.


What's the point of a SW satellite forum that has all the same rules as SW?

Share this post


Link to post
Share on other sites
Originally Posted By: Master Ackrovan
Quote:
This category includes graphics, spelling and grammar

Wouldn't Spelling and Grammar be in the Writing category? I mean, its got more to do with plot and atmosphere and such then it has to do with "Aesthetics".
Spelling and grammar doesn't have anything to do with plot and atmosphere. It does have to do with writing, though; and it also has to do with aesthetics. A well-written story with bad spelling is still a well-written story.

Share this post


Link to post
Share on other sites

I think that the reason we don't have enough reviewers, perhaps, is that the current setup is too time consuming. It takes a long time to sit down and write a meaningful review. The people who don't want to just write one line and put a number. We need more meaningful reviews, and if we want them we need a simpler system where you don't have a choice.

 

That said, I think that ADoS's idea sounded pretty good, so long as (like Acky suggested) spelling and typos are counted under writing. It makes absolutely no sense to keep it where ADoS had it under Aesthetics.

Share this post


Link to post
Share on other sites

Dikiyoba thinks everyone is making a far bigger deal out of "porn on SV" than it is. Whether you are for it or against it, it really does not happen that often.

Share this post


Link to post
Share on other sites

On the other hand, R-rated images do occur occasionally on the non Blades boards. I can see having an objection to them, especially if you are a parent with an impressionable child.

 

But, SV seems to be fairly well, and fairly regularly, moderated. If there was some need to keep the blades areas more entirely blades focused, I don't see any issue with execution.

 

But, there is still the overarching issue of making the CSR a better tool for increasing the audience for Blades scenarios. None of this chatter is going to make that happen without good examples. Forge had (or has) an example of how it might work, and I provided an example of one idea over on SV. I think it would be helpful, rather than to provide rubrics, to provide samples of how each person envisions the product. From there we can discuss how to make that happen. But, since the desire is to increase audience, and usefulness of the CSR, seeing an actual product will help us analyze effectively.

Share this post


Link to post
Share on other sites

I agree that we should focus on Blades for now, and put aside the squabbling

 

Originally Posted By: Master Ackrovan
Quote:
This category includes graphics, spelling and grammar

Wouldn't Spelling and Grammar be in the Writing category? I mean, its got more to do with plot and atmosphere and such then it has to do with "Aesthetics".

 

My rationale was that while spelling and grammar are techinically considered "Writing", they have literally nothing to do with the creative content of the scenario (which is the unifying characteristic of that rating category), but rather serve only to encourage or distract from immersion in said creative content, much like the rest of the "Aesthetics, Functionality, Etc." rating category. Perhaps the "Writing" title could be changed to "Creative Content". Of course, if the community strongly disagrees with spelling and grammar being where they are, it's not really a problem to switch it.

 

Quote:
Quote:
3. Linear<->Open-Ended (Can anyone think of a scenario where this might need an N/A? Not necessarily a BoX "scenario" but scenario in the literal sense.)

I'm not sure. IMAGINE THIS SCENARIO maybe? It's not really meant to be a "real" scenario anyway.

 

Then it should probably have a N/A option as well. I don't know what "IMAGINE THIS SCENARIO" is, but if it's anything like TM's "minimalism" for BoE, then it is indeed neither linear nor openended, but rather more like a single point.

 

Quote:
Quote:
Do the majority of BOX scenarios have a sufficient number of reviews for that to work?

Unfortunately not. Some scores for BoE CSR just give numbers, for instance. frown If this takes off, I suppose we'll have to try something, though. That might include a few reviews that just give scores we might have to knock off, and for BoE scenario's like the Little Girl we could just put all the numbers to zero, or as close as can be gauged by whoever is putting them up.

 

As I said, under this system, no reviews or ratings from CSR need to be discarded, even if we only have a score with no review.

 

But what does everyone think of making the three specific Rating categories and/or the three Genre ratings optional or mandatory? The Overall Rating clearly must be mandatory, but as for the rest, I think discussion is in order.

 

If we make them mandatory, we could probably figure out a way to port CSR without having to reanalyze each review to determine what the reviewer might have rated each category (I really hope to avoid that), but it might require a bit more thinking.

Share this post


Link to post
Share on other sites
Originally Posted By: The Almighty Doer of Stuff
Originally Posted By: Master Ackrovan
Quote:
This category includes graphics, spelling and grammar

Wouldn't Spelling and Grammar be in the Writing category? I mean, its got more to do with plot and atmosphere and such then it has to do with "Aesthetics".


My rationale was that while spelling and grammar are techinically considered "Writing", they have literally nothing to do with the creative content of the scenario (which is the unifying characteristic of that rating category), but rather serve only to encourage or distract from immersion in said creative content, much like the rest of the "Aesthetics, Functionality, Etc." rating category. Perhaps the "Writing" title could be changed to "Creative Content". Of course, if the community strongly disagrees with spelling and grammar being where they are, it's not really a problem to switch it.
I agree with this general principle; spelling and grammar is nothing to do with creativity. If people feel better to have the Writing category renamed to Creativity or whatever, then that's fine with me.

Share this post


Link to post
Share on other sites

I would support creating subforums for BoX reviews. I also have no problem with proxy postings in these forums for certain members like TM. Here is probably the best place for this stuff because it is the centralized hub for the community and will be for the future. No offense to any satellite communities.

 

As for review criteria and stuff, this was an issue back in the early community when all this came into being. There is no right answer here. Any scoring system inherently creates inconsistencies. Any rubric inherently favors certain types of scenarios over others. The best we could do was to simply create a basic score with comments that are public. The biggest complaint about the SW tables was that they were a black box with no accountability.

 

My suggestion is to leave the score with the review just like the CSR does. Everyone is going to have divergent opinions, but the averages are broadly meaningful. Just create a post requirement of 100 words per review or something like that so people cannot just treat it as a black box.

Share this post


Link to post
Share on other sites

The system I proposed doesn't really have a "rubric", per se. It splits up the quality ratings into three sections, but doesn't say on what you should base the number you put in each section. Also, for those scenarios that don't fit into those three sections, remember that the Overall Quality rating is independent.

Share this post


Link to post
Share on other sites

I'm a bit late here, and don't have any time to read or participate in the discussion (I am gone till September or so). However, I noticed that ADoS linked to my initial post on SV lampooning the CSR -- while omitting a link to the follow-up post in which I provided a statistical analysis of CSR and disproved several of the points I had been asserting in my lampoon. Therefore, if you read that first link, please consider also reading:

 

http://forum.nethergate.net/index.php?showtopic=1501

Share this post


Link to post
Share on other sites

Thank you Slarty. This is pretty good evidence the current system is working to a large degree. Could it be improved? Yes. Is it broken? I don't see any firm evidence for that.

 

No offense to anyone, but we've been through this before. Just about all of you were not around for it, but these same arguments came up. We were not able to reach a consensus then, but we pretty much agreed on a solution that, by and large, worked. As someone else said, the key is getting people to write cogent reviews that are helpful. The individual score is worthless, but the composites do provide some insight.

 

So here's my proposal and I can make it happen:

 

1) Blades of Avernum and Blades of Exile will have subforums for reviews.

2) Posts must be a review with a requirement of 100 words that make a good faith effort to provide how the reviewer feels about the scenario. A score from 1-10 must be provided with 1 being lowest and 10 being highest. Reviews must be legible and respectful and should not refer to any other reviews currently on the page. They should stand alone.

3) A scenario author may respond in a single post to a review to rebut any assertions made. Responses must be legible and respectful.

4) Posting is restricted and must be approved by a local moderator. Whether or not something gets posted is at the discretion of the moderator.

5) If a post is denied, the moderator must inform the person who posted with a statement of reasons. The poster may revise the comments and resubmit or appeal the decision to the board administrator. The administrator's decision is final.

 

More restrictive, but this is a good thing because it keeps things more structured. What do people think?

Share this post


Link to post
Share on other sites

I think splitting the score and not requiring people to write long reviews would produce more useful results in greater number than having a single score and requiring a long review for every review. Also, I like the idea of people being able to discuss the strengths and weaknesses of a scenario while reviewing it.

 

EDIT: Also, I still like the idea of the Blades Forge being a centralized resource for Blades. If it can get fixed up, then I think it would do that well.

Share this post


Link to post
Share on other sites
Originally Posted By: *i
3) A scenario author may respond in a single post to a review to rebut any assertions made. Responses must be legible and respectful.


To clarify, is this a single post per review?

Share this post


Link to post
Share on other sites
Originally Posted By: The Almighty Doer of Stuff
I think splitting the score and not requiring people to write long reviews would produce more useful results in greater number than having a single score and requiring a long review for every review.


One hundred words is not long at all. In fact, to show you just how short one hundred words is, I pieced together this little paragraph. I am certain that once you stop to think about how paltry a one hundred word review is, you will realize that it is not a significant barrier to people writing reviews. In fact, if you go and examine most of the existing reviews over at Shadow Vale, you will find that most of them already exceed this bare limitation. In my opinion, one hundred words, like this paragraph, is not difficult to write.

Share this post


Link to post
Share on other sites

Writing reviews shouldn't be difficult, though. Others may just want to post a short review, and not spend the better half of an hour typing something up. Also, short reviews are not bad; notice how much information you got in with your hundred words. It's things like four word 'This scenario really sucks' or one word 'Awesome' reviews that we're trying to avoid, and making people write entire dissertations on scenarios probably won't bring in any more reviews. Besides, it'd be a minimum. Nothing stops you from adding more.

 

This is also one hundred words. Would you say it's uninformative?

Share this post


Link to post
Share on other sites

This method seems fine with me. Many of my reviews are a little vague at the current CSR, I'll rewrite them to be in the hundred word mark. I still think that a "10-100" (which is really what we have) is clunky, so I will only award whole number scores on my reviews (and will try not to be obsessed with putting scenarios in 'order'), but I won't complain if everyone uses the decimal system we used before.

Share this post


Link to post
Share on other sites
Originally Posted By: *i
This is pretty good evidence the current system is working to a large degree. Could it be improved? Yes. Is it broken? I don't see any firm evidence for that.

I feel compelled to repeat myself. In my opinion, it's broken. Why? Because I can't find what I'm looking for when using CSR, primarily as a player.

Originally Posted By: *i
We were not able to reach a consensus then, but we pretty much agreed on a solution that, by and large, worked.

It didn't work for me. As a player, I felt that CSR failed me when I was trying to find things.

Originally Posted By: *i
What do people think?

I don't think it will fix the fundamental problem. However, I'm resigned to a system that I consider broken continuing indefinitely. So while I'd like to express again that I don't think this will help — at all — or even be a change, particularly, I don't expect anyone to do anything based on my opinion.

Share this post


Link to post
Share on other sites

The notion that posts must be approved by a moderator seems ... broken. Could you not, instead, put folks that write reviews into a group authorized to post there? Not that my reviews are great, or frequent, but they will be nonexistent if they need to be checked over by a third party prior to posting.

 

Also, it still doesn't address the issues which started this whole boondoggle.

 

How can we make the CSR a better tool for players, with the stated goal of EXPANDING THE PLAYERBASE!!!!

 

Instead of just making random changes for the sake of change, can we address that goal, even if in an infinitesimal way?

Share this post


Link to post
Share on other sites

What if, in addition to the normal review, they also could specify additional pre-set tags at the beginning/end of the review? Things like Combat Heavy, Linear, No Combat, Dungeon Crawl, and so forth. Nothing indicative of quality (that's what the part of the review and the score is for), just indicative of its attributes. Some of these could also be specified by the designer.

 

Then, in the first post of the scenario's thread, the number of times a tag appears is counted up, giving you a general idea of what the scenario contains. In scenario lists, tags that get marked off a few times (or once if by the designer) appear next to the scenario to give you an idea of what it contains.

Share this post


Link to post
Share on other sites
Originally Posted By: Naughty Salmon
Could you not, instead, put folks that write reviews into a group authorized to post there?

I'm not sure that making it so only an elite few can write reviews is a good idea. Sort of defeats the purpose of trying to get more reviews for the CSR, and it certainly won't expand the player base.

Now, it might be possible to have a group that can immediately post reviews and bypass third-party checking. That seems like it'd work for long-time and short-time reviewers.

Quote:
Also, it still doesn't address the issues which started this whole boondoggle.

How can we make the CSR a better tool for players, with the stated goal of EXPANDING THE PLAYERBASE!!!!

Instead of just making random changes for the sake of change, can we address that goal, even if in an infinitesimal way?

...Have you even read the thread?

EDIT: To clarify, I mean that moving it here is a step in the department of making it a better tool. Most SWers never wander out to SV or CRF, so having it here makes it more accessible. Improving the reviews themselves makes it better. So I don't see how you can say we aren't addressing that goal.

Share this post


Link to post
Share on other sites

Kel, the problem here is everyone is different and aggregate scores only capture average opinion. No rating system we could devise is going to be able to please everyone. Come up with every dimension or criterion you like. I'm sure somebody else would rather they be different. Point being, I do feel the current system is not broken any more than any other system we could employ. In fact, I would argue this is about the best we can do since simplicity is better in my mind.

 

The Blades Forge is an attractive idea and I want it to succeed. However, it will never have the sheer following of these forums. It's a great resource, don't get me wrong, but a vast majority of people will not even go there. This is not to be mean, this is just to be realistic. I don't think there is any better place for this stuff than here.

 

100 words is really not a lot. This is just meant to make people write something useful. They can write more if they like.

 

Salmon, I don't see what your beef is by having some quality control. It is no further effort on your part. You submit the review as a reply, moderator checks to ensure you aren't spamming and meet the very modest requirements, and approves it. I think the criteria for rejection should be pretty severe and happen only rarely. The only difference is your post does not appear immediately.

 

Believe me, my proposals are not random. In fact, there is no boondoggle. The current system worked when we had lots of people contributing and it will work again under those conditions. Will it work for absolutely everyone? Of course not and no system will. I honestly think 90% of the battle is visibility. SV, Blades Forge, etc. are completely off most people's radar. That's the problem I'm trying to tackle.

 

The reason I put the system in place is to keep the reviews clean. This does help players out. They can go to a thread, read everyones comments (the 100 word rule helps keep some quality), look at the scores, and make decisions on whether to play it.

 

That said, I don't think some elaborate CSR is going to expand the player base any more than a simple one. The key is getting a critical mass going. If people see activity, they want to be a part of it. The best way to show activity is to have something here. Anything else is just added complexity.

Share this post


Link to post
Share on other sites

I'm iffy on the "only letting certain people post" thing, but I do see the value in approving posts one-by-one.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0