Jump to content

Housekeeping: Revised Blades of Avernum Ratings System


Recommended Posts

http://forums.calref.co.cc/index.php?board=39.0

 

Many people have cited issues they have with the current Comprehensive Scenario Rankings system we have over at Shadow Vale. Some have argued for creating an entirely new system for rating and reviewing scenarios. Others say we should simply improve the current CSR to correct its failings. What do you think we should do? Have your say.

 

Voting has begun on the direction we should take the new (or simply modified) system, here. The poll will be restarted as new ideas are proposed, until the suggestions stop and we have a clear winner.

Link to comment
Share on other sites

  • Replies 385
  • Created
  • Last Reply

Top Posters In This Topic

Quote:
What's wrong with the CSR?


A couple of things. For one, its the reviews themselves. Most of them tend to be unhelpful to the designer or even the player (who is arguably the lesser important of the two and should therefore receive less attention). Commonly, a large amount of reviewers (including me, in all honesty)tend to be short, meaningless posts with a score. Reviewers would point out how amazingly technically advanced scenario's are (Blades of Rogue particularly) and hand it a score fractions over a 2. Most of the recent reviews have been more thoughtful, true, but its unlikely this trend will continue (I admit this is more of a personal opinion then fact). A fresh atmosphere is a good way to encourage people to "begin anew".

The second reason is that the current review system is too vague. Some reviewers start with 5's because they seem like the "fairest" number. Others start with 6 to take into account giving people "the benefit of the doubt", or some such like that. Other people (such as myself) just jamble all of the scenario in the review and try to pull a score out of their ass. Others have a fixed ratio and system (Lenar Lambs) and use that to decide a score. And the scores themselves aren't consistent. Some people consider a 5 to the the "average" like a C+ or something, while others (like myself) consider 7 to be the "average" score. There are plenty, plenty of other people that think differently.

As you can see, its really a mess. We need a fixed way to review a scenario that is both helpful to the player and the designer alike.

The Great Archon
Link to comment
Share on other sites

Originally Posted By: Master Ackrovan
A couple of things. For one, its the reviews themselves. Most of them tend to be unhelpful to the designer or even the player (who is arguably the lesser important of the two and should therefore receive less attention). Commonly, a large amount of reviewers (including me, in all honesty)tend to be short, meaningless posts with a score. Reviewers would point out how amazingly technically advanced scenario's are (Blades of Rogue particularly) and hand it a score fractions over a 2. Most of the recent reviews have been more thoughtful, true, but its unlikely this trend will continue (I admit this is more of a personal opinion then fact). A fresh atmosphere is a good way to encourage people to "begin anew".


I'd say the player is more important than the designer as far as reviews go, although there needs to be a good way for designers to get feedback as well. I'd also say scenarios that are technically advanced but not really fun to play should be getting low scores, although with BoR, from what I understand, it was getting very high scores for being complicated technically, despite not being particularly interesting from a player's perspective. I definitely agree with the comment about the fresh atmosphere, though, and the second paragraph is accurate.

We're building off threads on Shadow Vale about the shortcomings of the CSR, by the way. Here they are:
http://forum.nethergate.net/index.php?showtopic=1477
http://forum.nethergate.net/index.php?showtopic=1504

EDIT: As for why we need numbers, nobody wants to read through 400 threads to find the scenarios they might like. We need some way of searching more efficiently.
Link to comment
Share on other sites

Okay, regarding the vote kindly linked to by ADoS in the first post, So as to maintain security and prevent cheating from guest voting, registration is required to vote. you may do so by going here and filling out the form. You don't even have to do a stupid e-mail confirmation thing! (Also, for you paranoid types, you have an option to hide your mail address from registered members as well as guests).

 

Thanks,

w-dueck

 

Your local CRF admin type.

Link to comment
Share on other sites

You know, it really would make the most sense to either continue the discussion where it was (SV) or just hold it where the most people are (here).

 

Originally Posted By: Dintiradan
Reviewers send an e-mail to the designer, talking about the scenario in depth.

This is an interesting suggestion, but it would prevent designers from learning from the results of other designers' scenarios, unless designers made an effort to summarize and publish the feedback they received. I'm not saying that designers necessarily do get a lot of mileage out of the reviews they see of others' work, but it is something that the current system provides.

Link to comment
Share on other sites

Why not have two part reviews? The score could be based on the first part, which is entirely player perspective and all about fun and enjoyment. The second part could dig into the technical details and comment on scenario design. If you were really ambitious the second part could even have its own entirely different score!

 

—Alorael, whose feelings about reviews are that the numbers are usually unhelpful. The commentary is always where you can find the useful information. More text is always better.

Link to comment
Share on other sites

The discussion should obviously be here. SV and CRF are minor satellites. I would assume that everyone that has an interest in BOA comes here at least several times a month. A sticky should be made in the BOA section of the SW forums for the discussion. If we continue to host the discussion were only established and well known people will post then the result, I believe, will be at best neutral.
Link to comment
Share on other sites

I think this is probably the best place to continue the discussion, since it's the most neutral of the suitable sites.

 

Some ideas posed as questions that may or may not be useful:

 

-Would Aran be willing for anyone else to work on the Blades Forge? I understand that he's busy, but if the CSR has to go somewhere else, that would seem a good place for it to go.

-If the CSR is moved, might it be worth registering a separate domain name for it so that people don't object to whichever domain it is that it's on?

-Would a chat be useful to bash out some more developed ideas that could then be voted on or discussed later?

Link to comment
Share on other sites

Quote:
-Would Aran be willing for anyone else to work on the Blades Forge? I understand that he's busy, but if the CSR has to go somewhere else, that would seem a good place for it to go.


This was actually discussed in a chatroom. ADoS sent Arancaytar an Email about it. He hasn't gotten back to him yet.

Quote:
-If the CSR is moved, might it be worth registering a separate domain name for it so that people don't object to whichever domain it is that it's on?

Yes indeed

Quote:
-Would a chat be useful to bash out some more developed ideas that could then be voted on or discussed later?


There is CRF's chat, but that is of course not a good idea. If we all want to do this in a Chatroom, AIM would be the best bet. And the fairest.

Thoughts?
Link to comment
Share on other sites

I'd like an AIM chat as well, this whole thing is sorta confusing me.. the problem I have with the current CSR, is the ratings seem too specific. To me, I'm feeling, is there really much difference between a 7.5 and a 7.6. I have no problem with a number scale, but think there should be fewer rate options.

Link to comment
Share on other sites

Originally Posted By: Slith
If not, what is causing all the hubbub now?
1. People are unfamiliar with the formalities of rating a scenario.
2. There is a certain positive pressure to rate scenarios.
3. The current CSR location is problematic for some people.
4. There is an effort to gain participation in the BoX community, and many current community members see that the CSR in place is cumbersome and may, in fact, be an obstacle for new players.
5. ...
6. Profit.
Link to comment
Share on other sites

All this satellite bickering is a load of bollocks.

 

This balkanization, more than anything else, is the cause Blades has no healthy community anymore. And because the value of Blades is user-added, the lack of a healthy community means Blades never sold as well as it could have, which in turn made JV abandon the concept.

 

Now I'm seeing people compare this conflict to other historical conflicts, while others respond that this conflict is by far not as cool as the conflicts we used to get back in our day.

 

Whether Calamity Refuge versus Shadow Vale is like Polaris versus Desperance is not the point. The point is that Polaris versus Desperance was a broken concept. Polaris and Desperance are dead. They are dead like the Chance Forums are dead - the only difference is the nostalgia, and communities based on nostalgia don't work. If you want to live long as a satellite, you need to avoid these mistakes, not re-enact them.

 

...

 

Now that I've got this off my chest, I'll gladly participate in this discussion at any place you'll name. If we can agree on a scoring system, maybe it can even be integrated with the Blades Forge's scenario list.

 

Speaking of which, as I said to ADoS privately, I would very much like the next Blades Forge to become a team project, because I'm not up to doing that on my own again (witness that the Forge hasn't really been overhauled in almost two years). Anyone who messes about with PHP and Drupal would be welcome. smile

Link to comment
Share on other sites

My thoughts have largely been stated by other members.

 

Originally Posted By: Naughty Salmon
How is having a discussion of this type on three different forums helpful?

 

It isn't, duh.

 

Originally Posted By: VCH
The discussion should obviously be here. SV and CRF are minor satellites.

 

Wholeheartedly agreed. I feel like moving this thread.

 

Originally Posted By: Naughty Salmon

6. Profit.

 

Hear hear!

 

Originally Posted By: Arancaytrus
All this satellite bickering is a load of bollocks.

 

Aran wins at life. And this thread is moving. Now.

Link to comment
Share on other sites

Originally Posted By: Enraged Slith
Hurr hurr, we can't even agree on where to hold the discussion, much less what the problem is. Did people even have these discussions back during BoE's prime? If not, what is causing all the hubbub now?


Oh boy, did they ever. The word "rubric" was a running joke in the BoE community ever since a certain member made a ridiculously overcomplicated and lopsided checklist for giving every scenario an objective score out of 71. It was a bad idea then and it's a bad idea now.
Link to comment
Share on other sites

Originally Posted By: Arancaytar
Speaking of which, as I said to ADoS privately, I would very much like the next Blades Forge to become a team project, because I'm not up to doing that on my own again (witness that the Forge hasn't really been overhauled in almost two years). Anyone who messes about with PHP and Drupal would be welcome.

While I lack any particular skill to help, I would be excited to see that happen. (Although I stand by my offer to work on a desktop client to interface with part or all of the Forge.) One thing that I would personally be curious to learn more about is what the necessary work on the Forge actually entails; basically, what the project's development checklist looks like.
Link to comment
Share on other sites

The CSR's failing isn't the scoring system, or whether rubrics are used. The problem is that nobody bothers to write insightful reviews, and I don't think moving sites or adding categorical scores are going to solve that. The CSR has been the way it's been for years; it works when there's a healthy community of designers and players writing good reviews (see: geocities lyceum and most of ezboard Lyceum) and it sucks when the community is mostly disinterested (see: SV, or even worse the end of ezboard Lyceum.) I'm not convinced that any amount of rubrics will fix the CSR unless you fix Blades first, and good freaking luck with that.

 

While it would be cool to have ratings somewhere like the Blades Forge, I don't expect that to really fix anything and I couldn't demand any more of Aran than he's already done.

Link to comment
Share on other sites

We don't need a magical scoring system. We need good reviewers. We need lucid individuals who understand the capacity of Blades, the capabilities of the designer, the current level of innovation. We need communicators who will take the time to READ the scenario documentation and learn the author's purpose in design. They can then write a paragraph, and make an informed scoring decision on a scale of 0-10.

This can, and has worked. Unfortunately, it means some people shouldn't be reviewers. That is a harsh thing to have to tell someone, and this type of community, there is no one in a position to do it.

Link to comment
Share on other sites

If we had the web designer, we could honestly just set up a website with all the scenarios in a list with a little 5 star visual and people could just put the number of stars they thought a scenario earned as well as a comment section (a la Youtube). Visuals are easy to conceptualize and really cut into the heart of what our current system is. This would help players choose what they wanted to play, which, I think, is the most important aspect of the CSR. Most may want in-depth analyses, but those don't help anyone but the designer.

 

Designers should really look to beta testers for help and comments or ask specific questions. I know Nikki was more than happy to comment on my work in progress, so we should just pawn everything off on him.

 

Also, I'm pretty much in agreement with Laz (who needs to make another awesome scenario).

Link to comment
Share on other sites

I think that numbers are fine. It doesn't matter if people have different scales as long as their scales are internally consistent and they rate enough scenarios. The numbers provide a basis for what scenarios people read about - they choose the threads based on the scores, then choose scenarios based on the comments in the threads.

 

Wisdom of the crowds and all.

 

Move it to a "neutral" place, sure. But putting more rules/restrictions on reviewing isn't what is needed. Score and comments is the simplest effective method for reviewing scenarios.

 

DON'T provide any guidance for what a score means. Everyone has their own ideas of what a "9" scenario is. The point of Community (sure Comprehensive works too) reviews is that everyone's views are averaged into a sense of what scenarios are best.

Link to comment
Share on other sites

Salmon: Are you saying that if we ban people whose reviews aren't up to our exacting standards from reviewing scenarios, then people whose reviews we like will somehow decide to review more? I'm afraid I don't follow the logic in that. I imagine all that would happen is we'd have fewer reviews, and perhaps a rise in average quality, but no rise in max quality. I don't see why we can't have reviews open to everyone, and then have a separate place for in-depth discussion of the scenario. I imagine the people who don't know what they're talking about wouldn't stick around the in-depth discussions for too long, and if they do, there's no reason why you can't hit the "IGNORE" button if you don't like what they're saying.

 

As for reading the scenario documentation, I think that before we can enforce reviewers reading the documentation, we need to get on the cases of designers to keep their Read Mes small, and only containing the really important information. Version histories, hints, credits, etc. can all go in separate files. Even background info, like that in the readme for "Caverns of Stylbore Mountains", does not need to go in the Read Me. All that's necessary is a note that you should read Background.txt (or something like that) if you want to know what's going on.

 

The Read Me should, however, contain the intended purpose of the scenario, so people know they're not getting a plot-heavy adventure from, say, "Blades of Rogue", or "Sound Showcase". This does not mean, however, that they can't decide that the scenario isn't any good even though it fits its purpose exactly. Just because BoR succeeded in pushing the limits of Blades doesn't mean it was fun to play. Alternately, just because it didn't meet its stated purpose doesn't mean it isn't fun or otherwise useful for some other reason.

Link to comment
Share on other sites

Okay, so we don't need to standardize the exact meaning of each possible score, but shouldn't we at least insist that everyone gives a score out of 10? Or some other number?

 

Or alternatively, insist that they include what they're scoring out of; 9/10 is acceptable, 3/5 is acceptable, 92/100 is acceptable, but just 6 is unacceptable.

Link to comment
Share on other sites

I think, for searchability purposes, it would be best to have a standard scale of 0-10 or some other numbers or symbols or whatever. I agree with Drakey, though, that we should probably leave it vague as to what the numbers mean, except obviously the direction of the scale. For instance, if 10 is supposed to be a perfect scenario (however the reviewer defines "perfect"), and the reviewer loves the scenario but rates it 0, that does nobody any good.

 

Perhaps we could apply more well-defined scales to certain scales more than others, though. The quality scales (plot, gameplay, or whatever we have) should probably be vague, but genre scales (linear vs. open-ended, humorous vs. serious, etc.) should probably have a well-defined center, still leaving how far to either end the scenario is up to the reviewer.

 

I posted my review system idea on Calamity Refuge. If anyone wants me to post it here, I will (or someone else can copy and paste the whole post if they want), but I have to go catch a bus now.

Link to comment
Share on other sites

ADoS - While Channel 4 may interview some fans of Harry Potter on opening night of a film, it is Joyce Kulhawik that gives the reviews. There are a lot of folks here that could assume that role, and a handful that shouldn't. I'm saying that we should educate the handful (like a reviewing bootcamp) but not let them go primetime until they know what they are doing.

Link to comment
Share on other sites

Except can you really say that the "handful that shouldn't" are actually harming the review system in any way?

 

While there are obviously some reviewers that are better than others, the worst a bad reviewer does is add nothing. Those little plain-old one word "Bad/Good/Okay" followed by a score reviews do add something, however small. It helps further establish what the reviewers think of the scenario on a scale of quality, and while that may not be all that useful to the designer, it is very useful to the player when it comes to not wasting time with low-quality scenarios.

 

I think we'd all like every review to be sufficiently sized and detailed. But if it came down to having one or two detailed reviews per scenario, or a dozen one-word and a score reviews, I'd take the latter. I can always ask the reviewer what they did and didn't like about my scenario after the fact.

 

But in any case, I don't see how that's an issue. The average reviewer on the CSR right now tends to make reviews of sufficient length and detail. One problem we do have is that we don't have enough reviewers, and I can't see how turning away new ones would resolve that issue.

 

Thuryl: I can see where you're coming from, but that also poses a problem of having the review page cluttered with four pages of debate (think BoR). Why not just have a discussion page right next to it for that sort of thing? Keeps the review page clean and tidy, plus has the added benefit of allowing debates over the scenario.

Link to comment
Share on other sites

Originally Posted By: Nioca
Thuryl: I can see where you're coming from, but that also poses a problem of having the review page cluttered with four pages of debate (think BoR). Why not just have a discussion page right next to it for that sort of thing?


Because I think people are more likely to discuss a scenario when there are already others talking about it in order to review it than to initiate a discussion in an empty thread.
Link to comment
Share on other sites

Perhaps we should keep the reviews and the ratings separate but next to each other then? Have it so you don't have to enter a score to create a review, and vice versa.

 

Sort of like poll threads on message boards work, but designed better. You enter a thread for a given scenario, and at the top of the page is the categorical rating system. You can submit your rating without discussing/reviewing the scenario in words, or you you can discuss without rating. Spoilers are allowed but spoiler tags are strictly enforced.

 

EDIT: Also, you can see the results without voting, by default. This is important because chances are you're going to be picking a scenario based on the ratings first, and then, only after you have played it, will you place your own rating.

Link to comment
Share on other sites

Originally Posted By: Smiley-Faced Duck
I, for one, think the problem is that the CSR is hosted on Shadow Vale. I fully support moving it to somewhere — anywhere — else.


I think you're being bitter and unhelpful. I will admit that Shadow Vale has become incredibly unpleasant lately, but picking up and moving everything again is NOT the answer.

Good lord, I've said it before and I'll say it again. No more fragmentation of the community, please.

(doesn't make a very good battlecry, but oh well)
Link to comment
Share on other sites

Yeah, further bickering, like Ephesos said, will bring nothing but flames of misery and lead to a further divided BoA , which is NOT what we want to do at all. Besides, if anyone should be acting bitter, it should be me. And since I'm not publicly bitter, you have no right to be. tongue

 

But I do believe that moving the CSR should not be out of the question. It should be available to all of the community, regardless of ones personal feelings for one another.

 

I can't really participate in a AIM chat for a while (blame parents.) I'll have more to say on the rubric later.

Link to comment
Share on other sites

I still think we need some description of what a 1, 2, 3 or D, C, C+ is. Teachers don't mark without a list of letter grade descriptions, or one is available at the very least. I realise that everyone has a different idea of what outstanding or first class combat or plot is. Yet I still feel we need at least a sketch of what each letter or number actually means. We should be trying to standardise the reviewing process. Every reviewer should look at the list of number descriptions and then decide what number that scenario deserves. Otherwise it is like pulling a number out of the air. "I like 5s today, maybe a 5 will be a 7 tomorrow".

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...