Housekeeping: Revised Blades of Avernum Ratings System

[The Almighty Doer of Stuff]

Is that supposed to be a POX News joke? I agree with the sentiment, but it just isn't funny. :-P

[Acky]

Well, it wasn't planned. I just realized it after I wrote it.

[Lazarus.]

Well I likely won't be contributing anything to a new CSR. That's probably expected, since I've been contributing fairly little to the current CSR of late. I'd be willing to write reviews here, whenever I had anything worthwhile to say. But I'd also write reviews at SV.

 

Basically I'm not boycotting or condemning a move to Spidweb, but I don't care enough to help out.

[La paix]

Sheesh. Talk about esotericism! At least some people have more dedication than me. As Niemand said earlier, I too would love to see all this happen, but unfortunately, there's no way I can offer my services. Good luck.

[*i]

I will create the forums. I need to play around in the control panel to get things working correctly.

 

CM is correct. There will be no additional hassle other than moderators having to approve a post. Some people may find this elitist, but just about every serious publication has some oversight system. The point with this is I want the approval process to be transparent. There will be strict guidelines (that we should discuss) as to how a moderator decides on the review and offers means to revise the review. Not approving a review should be a rare circumstance.

 

As for the need, there may not seem to be one now. However, something like this would have been good when there was more activity on the CSR. It's been a long time, but I'm trying to take the long view here.

 

As for Kel's comment. I'm up for doing something different with scoring. Perhaps rate: "Poor, Mediocre, Fair, Good, Excellent" and then give a histogram?

[Nioca]

I'd go "Poor, Mediocre, Good, Excellent, Outstanding". But that's just me. But it raises a question: if we change the format, are we going to port old reviews over? And if so, how do we plan to make said ports compatible with the new format?

 

Anyway, I like that idea. It could work well. As for the histogram... why not just give a list of percentages for each opinion of the scenario. Like this:

Quote:

Reviewer Opinions of Uber-Awesome-Scenario

0/5 (0%) Outstanding

3/5 (60%) Excellent

1/5 (20%) Good

1/5 (20%) Mediocre

0/5 (0%) Poor

Simpler and easier to manage. And it could then be sorted into the five lists going by the mode review (so this would be an Excellent).

 

Taken a step further, we could add common opinions to it as well (Hard combat, so forth). Though maybe that's taking it too far.

[nikki.]

Two things.

 

1) stareye - does Jeff know about this, and is he happy to have the CSR here?

 

2) We already have a karma system for threads on the new UBB. Maybe those can be used to provide visual feedback as to how good a scenario is? So, for instance, a reviewer will submit a review, and then rate the topic using the 5 opinions stareye and Nioca spoke about above - 4 stars would be "Excellent", or whatever. That way, a player will be able to look at the listings and easily see which scenarios are more liked/better - everybody understand that 5 stars are better than 2.

[*i]

Lack of portability is the problem of going between two different systems. I don't really want to create some arbitrary translation. My inclination is to start fresh.

 

Jeff has pretty much told the moderator staff to run the forums. Adding a subforum is pretty non-invasive and helps encourage his products. It is no added effort for him and it supplants the existing feature on the Spiderweb site he has no time to maintain. He will be informed of the change, of course.

 

As for point 2, I will look into it. It will be tricky.

[*i]

The subforum is up. Please comment here on anything in the rules.

[The Almighty Doer of Stuff]

A scenario's author must request that a scenario be added, according to the rules. What about scenarios whose authors are no longer in the community? This has fewer implications for BoA, but for BoE that's the vast majority of scenarios that won't be in the system.

 

Also, will spoilers in reviews and responses be required to be in spoiler tags?

[Duck in a Top Hat]

Huzzah! It's fabulous! It's fantastic! I will go even further and say that it's fantabulous!

 

Seriously, though, I think it's great. The only problem which has already been mentioned is porting old reviews. It could be very, well, problematic. I'll volunteer to help in whatever I can.

[*i]

Actually, I believe I used the word "may" instead of "must" for requesting new scenarios. I clarified that moderators may also add new scenarios.

 

Good point about spoilers. What do people think?

 

I'd say we start over from a clean slate as far as reviews go. Keep links to the archival ones on Shadowvale, but new ones should go here.

[Ephesos]

My concern is that if we do not port reviews, we will never get anywhere, and will never reach the same amount of reviews we had before.

 

Also, on the topic of approving reviews, who among us will be approving them?

[*i]

I understand your point. Is there any credible risk of the past reviews just vanishing? Could they be preserved somewhere as historical? The point is to have access to that information so people can read them.

 

Right now I have it set for myself and any global moderator. I would like to have discussion here about that.

[*i]

Oh yeah, Nioca: I didn't use your set of words because "Outstanding" and "Excellent" are pretty much synonyms. If you gave me both of those words out of context and asked me which one is better, I would have a hard time picking one out. I think what I have is a bit more clear.

[Nioca]

I would make a better distinction between Rank 2 and 3. I generally think of "Average" and "Mediocre" being the same thing when it comes to ranking quality. Especially since "Average" can be interpreted as median, which is about the same as Mediocre. Maybe a "Poor, Mediocre, Good, Great, Outstanding"?

 

As for the Global Mod thing, I am a little concerned. Not with the actual mods, mind, that's fine. But I am concerned with how many of them would actually keep an eye on the board (I've counted three Global Mods that have an avested interest in BoA), and with the fact that all of them have responsibilities elsewhere as well. I think we need a couple low level regular mods, dedicated to the review forum and nothing else. They could stay on top of new developments in the BoA community without having to worry about the rest of the forums as well.

[*i]

Great and outstanding are still synonyms. I would rather the fifth tier be reserved for any exceptional scenarios and the fourth to be for ones of decent, but not very high, quality. The word great does not capture this.

 

Mediocre means unremarkable but generally has a negative connotation and is often synonymous with below average or inferior. I do think it is appropriate for this level.

[Enraged Slith]

I am a personal fan of "cowabunga".

[Ephesos]

Originally Posted By: Nioca

They could stay on top of new developments in the BoA community without having to worry about the rest of the forums as well.

This is more or less what I do already. Just sayin'.

[nikki.]

Poor, Mediocre, Average, Good, Excellent. Which is to say, what stareye already decided upon.

 

Edit (for more stuff): Also, Nioca, whilst there might be a flurry of posting activity at first, after that the review boards are never going to be busy enough to need more moderators than we currently have.

[*i]

I am open to suggestions on the words to use. My point with "great" versus "outstanding" is as follows: If I were to plot goodness versus word the two would come out pretty close.

 

If someone really wants to moderate the reviews forum, and generally meets the other good standing requirements of being a moderator, let me know.

[Swimmin' Salmon]

You do not have my permission to put my posts that exist in the CSR on SV into any forum on this message board.

[Ephesos]

Poor, Mediocre, Average, Good, Excellent...

 

Okay, here is my question. How on earth do you get an aggregate score out of that? Do you change the words to numbers? If so, how is that different from what we've been doing?

 

"This scenario is rated Good.543..."

[*i]

Quote:

You do not have my permission to put my posts that exist in the CSR on SV into any forum on this message board.

Will you at least write some of your reviews here and help out?

Technically we do not require anyone's permission. We can just link to it if we want.

But seriously. What are your objections? You are free to post pretty much whatever you want there. Really, what about the standards don't you like? I'm not seeing this hypothetical abuse considering we have guidelines that provide the appropriate check. What is to stop the moderators/administrators at SV from deleting things they don't agree with? Nothing. Your objections seem rather based on arbitrary principle.

Quote:

How on earth do you get an aggregate score out of that?

You don't. You get a histogram of what people thought. This comes down to whether we think averages on ratings are meaningful.

[nikki.]

Originally Posted By: *i

Technically we do not require anyone's permission. We can just link to it if we want.

Is that seriously a route anybody wishes to go down?

I suggest that we decide what kind of scoring system we're using, and fast. Yes, not everybody agrees with the 1-10 system we have at the moment, but I could list plenty of people who are against the suggested changes too. How about we keep the system we have, with an opt-out option - a designer can ask that his/her scenario is excluded from the scoring system but can still be reviewed (or excluded completely)?

Either way, if we don't decide something soon, which works for the majority of people, we're going to end up tearing the community apart, and frankly, it probably wouldn't survive that.

Addit: Of course, that's assuming that we haven't already seriously ticked off the handful of us who are still designing for, and playing, the game.

(Edit: changed the wording to be more family-friendly.)

[*i]

I was merely pointing out that we can link to anything else publicly available. I'm not suggesting we should. Big difference.

 

I'm up for whatever scoring system, or no scoring system. My one request is that it be simple.

[nikki.]

Originally Posted By: *i

I was merely pointing out that we can link to anything else publicly available. I'm not suggesting we should. Big difference.

Then I apologise, though the subtlety wasn't clear in the original post.

So, does anybody have a simple scoring system that will appease everybody?

[*i]

Sorry I didn't make it more clear. The "But seriously..." was supposed to do that.

 

Quote:

So, does anybody have a simple scoring system that will appease everybody?

 

Sadly, I don't think that is possible. If you assign numbers, that lends itself to problems. One is that there is no agreed definition of what a "7.2" means. Further, there is too much resolution when we have a decimal place -- reviews will probably be inconsistent in scoring. If you assign "qualifiers" then you avoid those problems, but it becomes very difficult to do any averaging.

 

I really cannot think of anything else like this.

 

One idea would be to get rid of the score and let reviewers have a "profile post" where they can discuss their favorite scenarios and their philosophy of what makes a good scenario. This might be the best alternative because you can find the reviewer(s) you most agree with and take their recommendations.

[The Almighty Doer of Stuff]

I think I like the scale you came up with, Stareye, except the word "mediocre" is rooted in "medium", and indeed means there's nothing really good about it, but there's nothing really bad about it either. Furthermore, calling the middle score "Average" could raise problems too, when you think about what the word really means, being a statistical mean.

 

How about Poor, Fair, Good, Very Good, and Excellent? "Fair" actually means what you wanted to say with "Mediocre" in this context, and is in the same place. Also, without using the terms meaning "median" and "mean" in the scale, it's okay for there to be more Poor and Fair scenarios than Good, Very Good and Excellent scenarios, which, for BoE at least, there are.

 

As for the "profile post", that sounds like an interesting idea, but it's not searchable at all, unfortunately. I think it would have to be a combination.

[Lilith]

Maybe we should just stick to thumbs-up or thumbs-down.

[Ephesos]

Okay... I have a proposal. This would, in theory, serve as the forum rules:

 

----------

 

Welcome to the Blades of Avernum Reviews forum! This forum is intended as a resource for both designers and players, so that players can decide what to play next, and designers can hone their skills.

 

Each scenario released for Blades of Avernum will have a thread created for it by a moderator. Authors may request that their scenario(s) be removed at any time, and it will be honored. If a scenario is removed, it may not be restored.

 

Review Guidelines

 

Remember, the main purpose of this forum is to help players and designers. As a consequence, please keep reviews constructive, but honest. One-line reviews don't help anyone, so try to offer more than just a final score. Of course, personal attacks and inappropriate language are grounds for a review's deletion by a moderator.

 

Scenario Scoring

 

With each review, we ask that you provide a numeric score between 1 and 10. This score helps us get a general idea of what scenarios are good, which are okay, and which are unplayable. Moderators will do their best to keep an up-to-date average score in the thread title for each scenario (PM one of us if something's wrong). Based on a scenario's average score, they will receive one of 5 labels:

• Excellent (10.0-8.0)
• Good (7.99-6.0)
• Average (5.99-4.0)
• Mediocre (3.99-2.0)
• Poor (1.99-0)

 

Author Responses

 

Scenario authors are entitled to respond to reviews of their work. Responses should stay in the scenario's thread, and authors should do their best to keep their posts clear and coherent (for example, don't respond to multiple reviews at the same time). As with the reviews themselves, please refrain from personal attacks and inappropriate language, because that will merit deletion.

 

----------

 

Okay. So, note two major things about this:

 

1. Post approval by moderator would be gone. I think people have generally behaved well enough in the past that we can eschew it. People who violate the CoC, of course, will be reprimanded.

 

2. Ye olde numeric system stays, but with the addition of categories. This gives a better indicator of what settles where (I hate this look anyway), and it mitigates the labeling of scenarios by numbers. It also allows us to keep the reviews that have already been written.

 

What do people think?

[*i]

Looks good. I would suggest trying to add that reviews should try to shoot for being 100 words if possible. This gives reviewers an artificial bar about what the helpful length is.

[Acky]

Very nice, Ephesos.

[Kelandon]

I much prefer Stareye's lower resolution. I might not avoid CSR if it were arranged with a 1-5 with no decimals. I'd still avoid it entirely if it stayed 1-10 with decimals.

[*i]

All right, what we could do is make a distinction between past and future reviews. Just translate things as Ephesos suggests. Future reviews can use a number (which gets translated to the coarse grained qualifier) OR they can simply provide the qualifier. Instead of doing averages, we can do a histogram.

[Ephesos]

I still dislike histograms. They add very little to what we already have.

 

Kel, if you don't like it, we could just say it has to be a round number. That seems reasonable. Past reviews would still translate as-is.

[*i]

Histograms provide more information. I don't think they adequately reflect the really like it versus really hate it crowd. Better people see the bimodal nature rather than something in the middle.

 

What is your specific issue with them?

 

EDIT: Point being, I don't see any reason to actually rank scenarios. If we skip that, there is no need for averaging.

[Ephesos]

I wager that most players would just get confused by the bimodal nature of the scores. Even if we just implemented something like dropping the outliers, we could mitigate the spread of scores without resorting to graphs.

 

EDIT: And if we don't rank scenarios, we should just scrap the freakin' CSR, drop the labels, eliminate numbers, and never say anything about a scenario that can be taken as offensive with relation to other scenarios.

[nikki.]

Originally Posted By: *i

EDIT: Point being, I don't see any reason to actually rank scenarios. If we skip that, there is no need for averaging.

Does anybody in the community agree with this? Or can we just assume that they do want to rank scenarios?

[Lilith]

Considering that the forum was originally called "Comprehensive Scenario Rankings", a system that doesn't actually rank scenarios seems perverse. I suspect the main mode of use for CSR is simply people wanting to play a good scenario and picking one from near the top of the ranking list.

[*i]

Does Amazon rank books? No. What would it even mean to do so? Sure, scenarios are a bit different, but I'm skeptical that saying scenario X is better than scenario Y because of a 0.06 point difference.

 

Perhaps this, listing scenarios by category based on the following:

 

Top Scenarios: At least 30% of review scores are rated tier 5.

 

Quality Scenarios: Does not meet top criteria, but at least 40% of reviews are tier 4 or better.

 

Worthy Scenarios: Meets neither other criteria, but at least 60% of reviews are tier 3 or better.

 

We can revise percentages however people feel fit. It's a little complicated to figure out how to categorize things, but it gives a pretty intuitive final result.

 

EDIT: I think this meaningfully sets apart the best scenarios but does not try to rank ones that are close in quality.

 

[Ephesos]

Originally Posted By: *i

Does Amazon rank books?

Yes.

[*i]

Not in the same way that we would do the rankings. But nowhere can you find a comprehensive ranking of all books ever written or sold. That's what I was implying.

[The Almighty Doer of Stuff]

That's by how much they sell, not how they're rated, though. We're not ranking scenarios by download numbers, after all.

 

Any thoughts on the Poor/Fair/Good/Very Good/Excellent scale?

[*i]

Seriously, let's give this thought. We've been ranking scenarios as we have for so long, it's difficult to think any other way. Recently, I've been wondering if this is even a useful exercise and if there might be better ways of suggesting scenarios for people to play.

 

Because what do people really care about from review scores? I contend that the list serves mostly to give people an impression of what is good and what is not. What do we as a community recommend? An aggregate score is one way of doing this, but not necessarily the most useful way.

 

What are people's thoughts on this?

 

EDIT: I suggested a threshold based approach. If a certain percentage of people liked a scenario it would make it on some list for recommendation. I do think this biases towards positive reviews, but I am personally fine with this. If a significant number of people really like something even though others may not be as fond, that means a player might as well and it would be a good idea to recommend it.

[The Almighty Doer of Stuff]

I like the idea a whole lot, personally. I don't think it's "perverse" at all, and I think it accomplishes the goal better than flat ranking can.

 

When it was just a flat placing into one of five categories, I was pretty ambivalent, but I like the threshold idea a lot.

[*i]

Here are a few thoughts. I think there is an inherent resolution to what humans can resolve when attempting to quantify something's quality. There is a reason so many places use a ranking of 1 to 5 stars, it is simple and people have a general feel for what the rankings mean. I propose taking it one step further and just naming 1 to 5 so we can get better agreement on what those mean.

 

To illustrate, I can't be precise to the first decimal place. Whether you got a 7.9 or an 8.2 really depended on how generous I was feeling at the time. If I had to score it again, you would probably get a similar, but different, score. What would not change, however, is my general impression of the scenario. For me, this scenario would be pretty good, but not great category. You would get this fairly consistent.

 

My second point is, are averages useful in this exercise? I think the answer is it depends. In the case where the scores fall along a bell curve, the average really is representative of general opinion. In cases where you have a significant number of people that like it and then a significant number that hate it, the average reflects no one's view of the scenario. In other words, the average is misleading. It is in these cases where some threshold based approach would provide more guidance.

[*i]

The other thing is if we go to a lower resolution we need to decide on what each number means. This is vital if we have any hope of accurately porting old reviews. I propose the following scale:

 

5 - Best

4 - Good

3 - Average

2 - Below Average

1 - Worst

 

We can name things differently when we decide what to call each score, but we should decide up front what each number means in broad terms.

[Wiz]

Originally Posted By: *i

I would rather the fifth tier be reserved for any exceptional scenarios

Why not name it 'Exceptional' then? This would also let you use 'Very Good' or 'Great' for 4 as suggested by some, without any ambiguity as to which is better.

[Swimmin' Salmon]

To answer your question, if you created a post on this forum based on one I wrote in either of the other CSR locations, I don't believe I would have the ability to edit my words. It would be unfortunate to have another set of reviews become off-limits to those that wrote them, just because it was decided to move the CSR to a new home, again. The last time was a necessity, due to the fragility of the ezboard structure, and the worry of loss of the CSR. This time, there is no such worry. I don't feel that I can approve of the heavy handed way this is happening. It seems one or two folks are heavily motived to make this happen, but the majority aren't.

 

So, why again? I remain unconvinced that this would be a beneficial change, and as it is being forced upon the majority, rather than voted upon by a majority, I have to be against the move.

 

At least take a poll of scenario designers.