Housekeeping: Revised Blades of Avernum Ratings System

[*i]

If you want to update your review that would be fine. Just make a new post and the old one can be made to go away. As long as we have moderators it should be fairly easy to update that. Certain members cannot post here directly, but I do support posting by proxy here because I think it is good for the community.

 

Really? Maybe we are talking to different groups of people. Most people I've spoken with have generally been supportive of moving it here. We get greater visibility and accessibility here, especially when it comes to people who don't know anything about Blades.

 

I would be supportive of a poll to find out what designers and established community members think. I'm not sure if a straight vote by majority is the right thing to base a decision from, but I do think it is good to at least get everyone's opinions and concerns aired and to try and address as many as possible.

 

EDIT: Just to clarify, I'm not sure just basing things off of 50% + 1 to be a good thing. If there is significant support (hard to put an exact number) and not a massive amount of vehement oppositon then we should move it here.

 

I am interested in hearing all of your concerns with moving it here. Hopefully we can address them or make a good effort to do so.

[Ephesos]

I'm perfectly alright with moving it here. It's starting from scratch here that worries me, when we're five years into working on a program which we know will never be updated or fixed again.

[The Almighty Doer of Stuff]

Question: What will we do with all the scores whose textual reviews were lost due to ezboard suckage? Drop them? Make special exceptions to the 100 word rule? Something else?

 

At any rate, the "make a new post and the old one will be deleted" idea seems very reasonable to me.

[keira]

Okay, I have an idea. It would run alongside whatever numeric rating we have in place, and would basically be an interface for users to 'tag' content.

 

Reviewers would be given a small list of basic traits of a scenario (such as combat-heavy, dramatic atmosphere, etc.). The reviewer would select what traits they believes belong to that scenario, and that's pretty much all they have to do.

 

Later, when a person looking for a scenario went to this site, they would be choose what they were looking for. So, if 'Bob' wanted a large-sized plot heavy humorous scenario, he would select 'Plot Heavy', 'Humorous', and 'Large' on a form and the site would filter out unwanted scenarios and pull up a list for him (The more options selected, the less results someone would get).

 

Limits would be put in place, however. x% of reviewers would need to say a scenario had a certain trait before it would qualify as that trait and be listed in the 'search' function. Also, scenarios would be prevented from being listed in the search function until x people had submitted tags.

 

This would be hosted on a completly third-party site (This is a little heavy for UBB, obvoisly) and links could be put into each scenario thread that takes users to a page for that scenario that shows it's tags (and how many voted for them).

 

 

 

Critique?

[*i]

I once thought this was a great idea a long while back. The problem is we tried them before with one of the contests a long while back. Issue is no one could agree on the definition of those terms whereas everyone can pretty much understand what the overall score is.

 

For example, your definition of "dramatic atmosphere" or "innovative" might differ from mine. This variance in terminological understanding makes things like this problematic. It's one thing when people disagree if a scenario has "quality X", it's quite another when no one can agree what exactly makes "quality X".

 

Also, simpler is better. Remember that real humans have to write reviews and real humans have to compile the results. The more complex the system, the greater the chance of error and the less chance of having people willing to actually do this. Also, having it span multiple sites just compounds the problem. I think we should keep things as central as possible.

 

[Ephesos]

I don't think we need a fancy tag cloud or anything for this. We can just have a separate thread of scenarios by type/genre.

[*i]

I don't think the 100 word rule will be hard anymore. Nonetheless, it is highly encouraged.

 

I agree that porting things is particularly thorny. It would be nice just to start over, but such is life. I do think we can effectively "translate" old reviews over to the new system. It won't be perfect, but it will probably do. The limiting factor is the effort required, especially on the BoE side.

 

I agree with Ephesos. Let's do our best to make something robust and good for the long term. That's why I'm advocating simple so people a five years from now have an easy system. As much as I hate UBB at times, this is probably the most stable place we are going to have for this. I don't envision these forums folding until Spiderweb Software does.

[Celtic Minstrel]

Originally Posted By: *i

Looks good. I would suggest trying to add that reviews should try to shoot for being 100 words if possible. This gives reviewers an artificial bar about what the helpful length is.

Or you could just say "write at least one good-sized paragraph" if you don't want to give a minimum word count. That still keeps the guideline without assigning a precise number to it.

[Kelandon]

Avoiding rankings would do much to alleviate my original problems with CSR. I used it exactly as Thuryl described, and it led me nowhere. It was only when I started ignoring the numbers and clicking randomly through the reviews and ignoring certain people's reviews that I found anything, but it was so slap-dash and haphazard that I found myself getting frustrated and stopped.

[Ephesos]

Kel, I'm sorry, but I really don't know what we can do to make you happy unless we nuke it all from orbit and make everyone rewrite every review they've ever written.

 

And I would also posit that some of your complaints have a lot more to do with small sample sizes.

[Lilith]

Originally Posted By: Kelandon

Avoiding rankings would do much to alleviate my original problems with CSR. I used it exactly as Thuryl described, and it led me nowhere. It was only when I started ignoring the numbers and clicking randomly through the reviews and ignoring certain people's reviews that I found anything, but it was so slap-dash and haphazard that I found myself getting frustrated and stopped.

This is, I suspect, the same problem that film critics face when reviewing films: they care a lot more about cinematography, innovation and all of that technical froofery than your typical cinemagoer, and they rate accordingly. On average, scenario designers (or at least, people involved in the design community) value different things in a scenario than ordinary players do, and scenario designers are also the people most likely to post lots of reviews. If the sort of people who write reviews simply don't agree with the sort of people who read them when it comes to what makes a good scenario, no rating system will solve that problem.

[Jerakeen]

Seconded.

Speaking as a very ordinary player, I found the reviews largely unhelpful, since I have to wade through a lot of opinions I might or might not agree with. All I really want to know before I start a scenario is:

 

1. Does it work?

2. Is it a party-building, quest-and-reward type of scenario, or some kind of arty one-off?

 

Technical innovation and artistic merit are all very well, but they don't necessarily make an enjoyable scenario.

[*i]

Agreed. Might I suggest we discuss a reviewer profile idea. That way players can read what reviewers look for in scenarios and can find one that matches their tastes?

[Kelandon]

Eph: In case you hadn't noticed, I was endorsing part of Stareye's plan. Not that hard to make me happy.

 

Turtle's point should be noted, as well. Rubrics suck, but something along the lines of ADoS's breaking out different categories might actually help a lot in addressing that players are looking for different things.

 

Then again, maybe what I'm really looking for is having a few of the bigger, HTML reviews that (I think) pre-dated CSR, in addition to the CSR reviews. Not sure. Eh. Not that important.

 

Another thing that I've regularly found frustrating, mostly as a designer but to some extent as a player as well, is that people tend to be rating the Beta or the 1.0.0 version. I've never left scenarios at 1.0.0; the HLPM has been through upwards of 30 versions at this point, and Bahs, LP, and Exodus were all dramatically modified in response to player comments. I don't know if there's anything we can do to address this, except maybe adding a rule that people need to indicate which version it is that they're rating, but I thought I might bring it up anyway in case someone else has a better idea.

[Acky]

I agree with Kelandon from the players perspective on the Beta reviews. It's very aggravating reading what the Beta was, and not the final product, which is what I will be playing. I don't want a *rule* to be in place, but I don't think many people will object to requesting common sense.

[Lilith]

Originally Posted By: Kelandon

Then again, maybe what I'm really looking for is having a few of the bigger, HTML reviews that (I think) pre-dated CSR, in addition to the CSR reviews. Not sure. Eh. Not that important.

I don't think this would be a bad thing, but keep in mind that only a few scenarios ever had those reviews, and most that had them only had one.

[The Almighty Doer of Stuff]

They're still up, if anyone wasn't aware.

http://es.geocities.com/mabplay/alcritas_lyceum/scenrev.html

 

I don't know how long that will last, however, given the impending doom of Geocities.

[Celtic Minstrel]

Yes, we should probably require that you specify which version of the scenario is being reviewed.

[Ephesos]

Originally Posted By: Celtic Minstrel

Yes, we should probably require that you specify which version of the scenario is being reviewed.

...Um. I posit that this is unnecessary. A gentle reminder post from the author or a moderator that reads "please download the newest version" should suffice.

And I'm really starting to like the thumbs up/down idea more. Porting could be as simple as 5.0 and above is a thumbs up, below is down. Maybe leave a "meh" category for middle-of-the-road ratings.

I had kind of a weird reaction when I realized that I (quite recently) rated A Visit to the Madhouse higher than Valley of Dying Things. I mean, I really enjoyed both, and both are quite well-made. So that difference in the scores started to feel trivial.

...ah, forget it. I'm going to go back to actually working on a scenario now.

[nikki.]

Originally Posted By: Ephesos

...ah, forget it. I'm going to go back to actually working on a scenario now.

Thumbs up.

[Kelandon]

Originally Posted By: Ephesos

Originally Posted By: Celtic Minstrel

Yes, we should probably require that you specify which version of the scenario is being reviewed.

...Um. I posit that this is unnecessary. A gentle reminder post from the author or a moderator that reads "please download the newest version" should suffice.

Huh? How does reminding people that they should get the latest version help anything? After all, if I'm a player, and I'm reading a review from 8/12/09 but today happens to be 8/12/10, I may have the latest version but still can't tell for the life of me whether the review is for the version I've got or not unless the review specifies.

[Ephesos]

Alright, fine. Answer this then, Kel: How many people actually pay attention to the damn version number?

[tridash]

Why not just make a post in the review thread when a new version is released? All reviewers after that should be reviewing that version unless they clearly state otherwise.

[Celtic Minstrel]

That sounds reasonable to me. The "new version ready" post could be made either by the author or by a moderator.

[Ephesos]

Thank you tridash, for more effectively stating what I meant in my post.

[Kelandon]

I s'pose that would be fine.

 

Though, as Eph has pointed out, most people DON'T pay attention to the version numbers and as a result keep old versions around. (I'm recalling a discussion we had that went something like, "What? You mean there's a version of the HLPM after v1.1?") So this wouldn't necessarily be a perfect fix, but I'm not sure that it's a big enough concern that we need a perfect fix.

[*i]

All right. Let's move ahead with this because if we don't need anything nothing is going to happen.

 

Version numbers should be included and some post should be made by the moderator or author to inform people that the scenario has been updated.

 

We can create some categories as far as genre. I think we should leave it to the author to assign them. If reviewers disagree, they may direct complaints within the reviews and adjust scores accordingly.

 

I'm think a 1 to 5 integer scoring system is probably the best we can do. I do like thumbs up and thumbs down, but I do like having something that distinguishes between good and the best.

 

I'm looking to start the pilot program of this on a few scenarios this weekend. Please offer any ideas.

[Nioca]

I'll freely offer up Witch Hunt, if that's what you mean.

[Duck in a Top Hat]

Assuming that is what you mean, you can take Turn That Frown.

[*i]

Thanks. Yes. Volunteers for scenarios would be most appreciated.

[tridash]

Feel free to use Incorruptible if you need another short scenario.

[*i]

I have a thread for APF, one of my scenarios from half a decade ago. Please give some reviews to test this.

 

For those who asked me to use your scenario, thanks! Could you please get me:

 

Difficulty

Version number

Keywords

 

See my APF post for help coming up with these.

[BainIhrno]

Will we ever have BoE scenarios up?

[nikki.]

Request: Can reviewers please not let the engine differences between BoA/BoE bias their reviews?

[Ephesos]

I'll second that request.

[Nioca]

Originally Posted By: *i

For those who asked me to use your scenario, thanks! Could you please get me:

Difficulty
Version number
Keywords

I'm guessing Difficulty refers to the level recommendation, not the actual difficulty of the scenario. Either way:

Difficulty - Beginner
Version Number - 1.0.3
Keywords - Linear, Combat Heavy, Avernum Universe, Short

[Dikiyoba]

Nioca, if your review needs a summary, then it's too darn long.

 

Dikiyoba.

[Ephesos]

...yeeeeeah. Agreed.

[Duck in a Top Hat]

Turn That Frown Upside-down

 

Difficulty - beginner

Version Number - 1.0

Keywords - smiles, frowns, beginner, short, linear

[*i]

Every reviewer is going to have his or her own style. Let's try not to criticize people. Such detail should be welcomed. Thanks Nioca.

[*i]

Yes. We will do BoE as well. I just wanted to test it with BoA first.

[The Almighty Doer of Stuff]

If you're too impatient to read a long review, then don't. Nobody's forcing you to do so. Your loss if the review says something insightful that would determine whether you'd play the scenario. "tl,dr" isn't a helpful comment and you know it.

 

Don't be discouraged, Nioca.

 

I'd volunteer my scenarios, but they're only for BoE. When you create the forum, though, I'll give the details.

[Swimmin' Salmon]

If someone is commenting that a review is too long, aren't they saying it wasn't helpful? I thought Diki was giving open and honest feedback on a process which was in flux. Therefore, I don't really understand the compulsion to berate her for at least contributing something. My understanding is that this change is being designed to help a player find something which they would like playing. Shouldn't the requirements of reviews reflect that goal? I'll withhold judgment for the time being.

[Kelandon]

I prefer a longer reviews, myself. If you don't want to read the whole thing, just read the summary.

[Celtic Minstrel]

I disagree with Dikiyoba/Ephesos - I rather like the detail that Nioca gives.

[keira]

Long reviews are better than short reviews. Remember that the reader can always skim it, but it's hard to reverse this for short reviews.

[Dikiyoba]

I should clarify that it's not the review's length in and of itself that's the problem. If the review was long but all of it was interesting, I wouldn't have said anything. But the review is only long because it's repetitive. The summary, the notable points, and the rest of it all cover basically the same thing, so everything gets said at least three times. That makes for a pretty boring review, especially since it's unlikely that the review will be radically different from the other reviews.

 

In short, make sure nothing is getting said over and over again and the review will be a lot more fun to read even if it is still long. (Well, I'll enjoy reading it a lot more, anyway.)

 

---

 

Originally Posted By: Salmon

Therefore, I don't really understand the compulsion to berate em for at least contributing something.

In all fairness, most of what I contribute is probably complaints.

 

Also, Dikiyoba fixed your typo.

[*i]

Some people find longer, in depth comments more helpful. Others don't. We're not going to find two players alike and it's impossible to please everyone. Rather, if you don't like the way a review is done, make your own. That way everybody wins.

 

One thing I do know to be counterproductive is criticizing people who are just trying to help. Sure, offer suggestions on how things can be improved. That's fine. However, it might be best to do so privately.

[*i]

More scenarios are always welcome!

 

Just need the scenario name, recommended levels, version, and some keywords. Consult the thread in the Rankings subforum for assistance there.

[nikki.]

The Eternal

Levels 10-15

Version 1.0.0

Short, Serious, Mystery, Linear, Light/No Combat, Dark, Plot Heavy.