Jump to content


Housekeeping: Revised Blades of Avernum Ratings System


  • Please log in to reply
385 replies to this topic
The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,402 posts
  • LocationMassachusetts

Posted 22 July 2009 - 07:09 AM #36 Housekeeping: Revised Blades of Avernum Ratings System

Salmon: Are you saying that if we ban people whose reviews aren't up to our exacting standards from reviewing scenarios, then people whose reviews we like will somehow decide to review more? I'm afraid I don't follow the logic in that. I imagine all that would happen is we'd have fewer reviews, and perhaps a rise in average quality, but no rise in max quality. I don't see why we can't have reviews open to everyone, and then have a separate place for in-depth discussion of the scenario. I imagine the people who don't know what they're talking about wouldn't stick around the in-depth discussions for too long, and if they do, there's no reason why you can't hit the "IGNORE" button if you don't like what they're saying.

As for reading the scenario documentation, I think that before we can enforce reviewers reading the documentation, we need to get on the cases of designers to keep their Read Mes small, and only containing the really important information. Version histories, hints, credits, etc. can all go in separate files. Even background info, like that in the readme for "Caverns of Stylbore Mountains", does not need to go in the Read Me. All that's necessary is a note that you should read Background.txt (or something like that) if you want to know what's going on.

The Read Me should, however, contain the intended purpose of the scenario, so people know they're not getting a plot-heavy adventure from, say, "Blades of Rogue", or "Sound Showcase". This does not mean, however, that they can't decide that the scenario isn't any good even though it fits its purpose exactly. Just because BoR succeeded in pushing the limits of Blades doesn't mean it was fun to play. Alternately, just because it didn't meet its stated purpose doesn't mean it isn't fun or otherwise useful for some other reason.
The unofficial Blades of Exile merchandise shop: No markups, cheap prices, quality printing! Spread the word!

Celtic Minstrel Celtic Minstrel

Eye of Avadon

  • Global Moderator
    • Member ID: 5,296
  • 3,868 posts

Posted 22 July 2009 - 07:20 AM #37 Housekeeping: Revised Blades of Avernum Ratings System

Okay, so we don't need to standardize the exact meaning of each possible score, but shouldn't we at least insist that everyone gives a score out of 10? Or some other number?

Or alternatively, insist that they include what they're scoring out of; 9/10 is acceptable, 3/5 is acceptable, 92/100 is acceptable, but just 6 is unacceptable.
Exile 2 Book of Items (I didn't create it, I just converted it to HTML.)

"Man, I know how you feel. I once spent an hour playing WordPad before realising that it was a text editor." – Thuryl

"Dikiyoba just hopes no one ever blows up Saturn. Getting those rings back into their proper position would take hours."

"—Alorael, who spells phoenetically. No matter how much his orthography is a wreck, intelligibility rises from the ashes."

"I've never played a Spiderweb game. I didn't even know SW made games until I saw this thread." —Nikki

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,402 posts
  • LocationMassachusetts

Posted 22 July 2009 - 07:43 AM #38 Housekeeping: Revised Blades of Avernum Ratings System

I think, for searchability purposes, it would be best to have a standard scale of 0-10 or some other numbers or symbols or whatever. I agree with Drakey, though, that we should probably leave it vague as to what the numbers mean, except obviously the direction of the scale. For instance, if 10 is supposed to be a perfect scenario (however the reviewer defines "perfect"), and the reviewer loves the scenario but rates it 0, that does nobody any good.

Perhaps we could apply more well-defined scales to certain scales more than others, though. The quality scales (plot, gameplay, or whatever we have) should probably be vague, but genre scales (linear vs. open-ended, humorous vs. serious, etc.) should probably have a well-defined center, still leaving how far to either end the scenario is up to the reviewer.

I posted my review system idea on Calamity Refuge. If anyone wants me to post it here, I will (or someone else can copy and paste the whole post if they want), but I have to go catch a bus now.
The unofficial Blades of Exile merchandise shop: No markups, cheap prices, quality printing! Spread the word!

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 22 July 2009 - 08:11 AM #39 Housekeeping: Revised Blades of Avernum Ratings System

ADoS - While Channel 4 may interview some fans of Harry Potter on opening night of a film, it is Joyce Kulhawik that gives the reviews. There are a lot of folks here that could assume that role, and a handful that shouldn't. I'm saying that we should educate the handful (like a reviewing bootcamp) but not let them go primetime until they know what they are doing.
Got pr0n? Join the Spiderwebbers!

Lilith Lilith

in before the apocalypse

  • Global Moderator
    • Member ID: 669
  • 19,828 posts
  • LocationMelbourne, Australia

Posted 22 July 2009 - 02:42 PM #40 Housekeeping: Revised Blades of Avernum Ratings System

Part of the problem is that CSR's structure discourages any actual debate about scenarios: posts that don't contain a score are subject to deletion, so everyone is basically allowed one post per scenario.

Nioca Nioca

Heart of Avadon

  • Member
    • Member ID: 4,702
  • 4,554 posts

Posted 22 July 2009 - 03:02 PM #41 Housekeeping: Revised Blades of Avernum Ratings System

Except can you really say that the "handful that shouldn't" are actually harming the review system in any way?

While there are obviously some reviewers that are better than others, the worst a bad reviewer does is add nothing. Those little plain-old one word "Bad/Good/Okay" followed by a score reviews do add something, however small. It helps further establish what the reviewers think of the scenario on a scale of quality, and while that may not be all that useful to the designer, it is very useful to the player when it comes to not wasting time with low-quality scenarios.

I think we'd all like every review to be sufficiently sized and detailed. But if it came down to having one or two detailed reviews per scenario, or a dozen one-word and a score reviews, I'd take the latter. I can always ask the reviewer what they did and didn't like about my scenario after the fact.

But in any case, I don't see how that's an issue. The average reviewer on the CSR right now tends to make reviews of sufficient length and detail. One problem we do have is that we don't have enough reviewers, and I can't see how turning away new ones would resolve that issue.

Thuryl: I can see where you're coming from, but that also poses a problem of having the review page cluttered with four pages of debate (think BoR). Why not just have a discussion page right next to it for that sort of thing? Keeps the review page clean and tidy, plus has the added benefit of allowing debates over the scenario.

Lilith Lilith

in before the apocalypse

  • Global Moderator
    • Member ID: 669
  • 19,828 posts
  • LocationMelbourne, Australia

Posted 22 July 2009 - 03:20 PM #42 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Nioca
Thuryl: I can see where you're coming from, but that also poses a problem of having the review page cluttered with four pages of debate (think BoR). Why not just have a discussion page right next to it for that sort of thing?
  

Because I think people are more likely to discuss a scenario when there are already others talking about it in order to review it than to initiate a discussion in an empty thread.

Duck in a Top Hat Duck in a Top Hat

Shadowwalker

  • Member
    • Member ID: 6,265
  • 514 posts

Posted 22 July 2009 - 04:40 PM #43 Housekeeping: Revised Blades of Avernum Ratings System

I, for one, think the problem is that the CSR is hosted on Shadow Vale. I fully support moving it to somewhere — anywhere — else.

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,402 posts
  • LocationMassachusetts

Posted 22 July 2009 - 04:43 PM #44 Housekeeping: Revised Blades of Avernum Ratings System

Perhaps we should keep the reviews and the ratings separate but next to each other then? Have it so you don't have to enter a score to create a review, and vice versa.

Sort of like poll threads on message boards work, but designed better. You enter a thread for a given scenario, and at the top of the page is the categorical rating system. You can submit your rating without discussing/reviewing the scenario in words, or you you can discuss without rating. Spoilers are allowed but spoiler tags are strictly enforced.

EDIT: Also, you can see the results without voting, by default. This is important because chances are you're going to be picking a scenario based on the ratings first, and then, only after you have played it, will you place your own rating.
The unofficial Blades of Exile merchandise shop: No markups, cheap prices, quality printing! Spread the word!

Ephesos Ephesos

A God in Mote's Eye

  • Moderator
    • Member ID: 2,749
  • 6,157 posts

Posted 22 July 2009 - 04:50 PM #45 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Smiley-Faced Duck
I, for one, think the problem is that the CSR is hosted on Shadow Vale. I fully support moving it to somewhere — anywhere — else.


I think you're being bitter and unhelpful. I will admit that Shadow Vale has become incredibly unpleasant lately, but picking up and moving everything again is NOT the answer.

Good lord, I've said it before and I'll say it again. No more fragmentation of the community, please.

(doesn't make a very good battlecry, but oh well)

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 22 July 2009 - 05:36 PM #46 Housekeeping: Revised Blades of Avernum Ratings System

Yeah, further bickering, like Ephesos said, will bring nothing but flames of misery and lead to a further divided BoA , which is NOT what we want to do at all. Besides, if anyone should be acting bitter, it should be me. And since I'm not publicly bitter, you have no right to be. tongue

But I do believe that moving the CSR should not be out of the question. It should be available to all of the community, regardless of ones personal feelings for one another.

I can't really participate in a AIM chat for a while (blame parents.) I'll have more to say on the rubric later.

Dikiyoba Dikiyoba

Heart of Avadon

  • Member
    • Member ID: 4,222
  • 8,999 posts

Posted 22 July 2009 - 05:51 PM #47 Housekeeping: Revised Blades of Avernum Ratings System

No location will work for absolutely everyone. (And it's not like there aren't indirect methods if someone has something they really want to say.)

Dikiyoba.

VCH VCH

Eye of Avadon

  • Member
    • Member ID: 210
  • 3,485 posts

Posted 22 July 2009 - 06:15 PM #48 Housekeeping: Revised Blades of Avernum Ratings System

I still think we need some description of what a 1, 2, 3 or D, C, C+ is. Teachers don't mark without a list of letter grade descriptions, or one is available at the very least. I realise that everyone has a different idea of what outstanding or first class combat or plot is. Yet I still feel we need at least a sketch of what each letter or number actually means. We should be trying to standardise the reviewing process. Every reviewer should look at the list of number descriptions and then decide what number that scenario deserves. Otherwise it is like pulling a number out of the air. "I like 5s today, maybe a 5 will be a 7 tomorrow".
Мама, я в России!

Nioca Nioca

Heart of Avadon

  • Member
    • Member ID: 4,702
  • 4,554 posts

Posted 22 July 2009 - 06:45 PM #49 Housekeeping: Revised Blades of Avernum Ratings System

Something like this?

EDIT: Changed into a link.



Niemand Niemand

Hand of Avadon

  • Moderator
    • Member ID: 3,580
  • 2,138 posts
  • LocationWisconsin

Posted 22 July 2009 - 06:55 PM #50 Housekeeping: Revised Blades of Avernum Ratings System

I haven't yet taken the time to read all of the text in every cell, but that looks pretty good to me.
Possibly you suppose that [he] is a harmless fellow, who with vacant mind recites some empty formula,
traces a wandering sign in the air, and with palsied hand shakes a wand the while he gibbers his insanity at the yawning moon.

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 22 July 2009 - 07:16 PM #51 Housekeeping: Revised Blades of Avernum Ratings System

Quote:
No location will work for absolutely everyone. (And it's not like there aren't indirect methods if someone has something they really want to say.)

Dikiyoba.

But that's something that should be avoided, not counted on. And true, one could just Email the review to the designer himself, but then what use is it to the player and/or anyone else

Quote:
I still think we need some description of what a 1, 2, 3 or D, C, C+ is. Teachers don't mark without a list of letter grade descriptions, or one is available at the very least. I realise that everyone has a different idea of what outstanding or first class combat or plot is. Yet I still feel we need at least a sketch of what each letter or number actually means. We should be trying to standardise the reviewing process. Every reviewer should look at the list of number descriptions and then decide what number that scenario deserves. Otherwise it is like pulling a number out of the air. "I like 5s today, maybe a 5 will be a 7 tomorrow".

Second. Like I mentioned earlier, this is the state of things. If not for the direct benefit of the Designer, the Player would benefit at least from a consistent and sensible rubric.

Drakefyre Drakefyre

Manifest Destiny

  • Member
    • Member ID: 8
  • 9,598 posts

Posted 22 July 2009 - 07:59 PM #52 Housekeeping: Revised Blades of Avernum Ratings System

I disagree. As long as reviewers try to be internally consistent, it should be fine. Wisdom of the crowds, law of large numbers, etc.

We could also provide median scores in addition to mean scores.

EDIT: And yes, specifying a 1-10 scale or 0-10 scale should be done. But aside from saying that a reviewer should "like" a scenario they rate 7 more than they "like" a scenario they rate 6.5, we don't need much. ("Like" or "think is better than" both work)

Although we should discourage vindictive/adjusting ratings that are exaggerated to shift the mean.

EDIT 2: Why couldn't we host CSR on these boards? A forum right next to the 3 Blades forums.
“Iron rusts from disuse; water loses its purity from stagnation ... even so does inaction sap the vigour of the mind.” - Leonardo da Vinci
====
Drakefyre's Demesne  - Happy Happy Joy Joy
Encyclopedia Ermariana  - Trapped in the Closet
====
You can take my Mac when you pry my cold, dead fingers off the mouse!

Enraged Slith Enraged Slith

Revenge of the Slith

  • Member
    • Member ID: 27
  • 2,634 posts

Posted 22 July 2009 - 09:19 PM #53 Housekeeping: Revised Blades of Avernum Ratings System

I'm pretty sure Jeff has said no to this many times.

Callie Callie

SuperCalliefragilisticexpialidocious

  • Member
    • Member ID: 4,904
  • 4,168 posts
  • LocationNevada, USA

Posted 22 July 2009 - 10:28 PM #54 Housekeeping: Revised Blades of Avernum Ratings System

I agree with Drakefyre.  I don't see the need to establish a rubric or any other concrete method of scoring.  Having recently read some reviews, I noticed that all of the reviews offered useful points to the designer, yet there was a variety of methods used to score the scenario, and all of the methods seemed to produce a reasonable conclusion.  Plus, some people will likely be deterred by having to conform to a set guideline.
"I'm so glad you mentioned [goat heads] were weeds. I had never heard of them before, and thought maybe Excalibur lived in an area strewn with the heads of slaughtered goats. : p " - Tyranicus
Restless Souls - My first BoA scenario

Ephesos Ephesos

A God in Mote's Eye

  • Moderator
    • Member ID: 2,749
  • 6,157 posts

Posted 22 July 2009 - 10:57 PM #55 Housekeeping: Revised Blades of Avernum Ratings System

I agree with Drakefyre, and think that just hosting the reviews here in another forum would minimize all of this fuss. Subforum of BoA and all that. I'd be happy to moderate that.

Also, that table is intimidating, but I suppose it works.

Lazarus. Lazarus.

Hand of Avadon

  • Member
    • Member ID: 3,972
  • 1,223 posts

Posted 22 July 2009 - 11:21 PM #56 Housekeeping: Revised Blades of Avernum Ratings System

I'd be fine with moving the CSR here, but I always assumed that Jeff objected (because spiderweb had that terrible review section on its terrible scenario tables.)

If this isn't the case, or if Jeff's opinion is different now that they don't even pretend to keep up the tables, then cool. I don't know why it didn't come up when we were moving the CSR to SV in the first place....

Edit: Also it could be a subforum here to avoid cluttering the main page.
"It's all that weiner kid TM's fault." -Nikki

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 23 July 2009 - 04:39 AM #57 Housekeeping: Revised Blades of Avernum Ratings System

Here? I honestly don't believe that Jeff will allow the CSR on his own website. He doesn't really support the Blades Community as it is, so what makes you guys think he'll put it up? But, hey, it makes everyone happy, so its definitely worth a shot. At worst, he'll just say no, and we would be no further back then before.

(also, we should probably ask him relatively soon, because he's supposed to start Beta testing A6 in a few weeks, and will be more likely to just say no out of annoyance then anything else.)

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,402 posts
  • LocationMassachusetts

Posted 23 July 2009 - 06:52 AM #58 Housekeeping: Revised Blades of Avernum Ratings System

The Blades Forge seems like a pretty neutral location to me, really. Does anyone have some strong dislike for Arancaytar or Ermarian.net? Plus, unlike a message board-based system like the current CSR, a Blades Forge-based system has more flexibility and searchability.

I was talking to Bain-Ihrno last night, and we thought of some ideas. Perhaps, in addition to the "poll thread"-style rating/review/discussion system, we could have a button next to the results that says "Ratings By User" that would display each individual rating and the person who rated it that way. Perhaps you could even choose to eliminate the ratings of certain people who you tend to disagree with, or search only for ratings by people you tend to agree with. This is better than the thumbs up/down system that was proposed once, because it's more personalized to the user and doesn't make other reviewers who might get too many thumbs down unjustly feel like they're being bullied.

As for Nioca's rubric, I find myself disagreeing with many aspects of it. This is one of the problems with the huge rubric that Drakey mentioned: not everyone agrees with what makes an aspect of a scenario good and what doesn't. In addition, that's a LOT of reading to expect everyone who tries to use the system to read before they can rate scenarios.

While I don't think we should make the system overly simple to the point where it's not functional or practical, I strongly believe we should make it as easy as possible to use the system. Complex rubrics, while useful in theory, will tend to just discourage people from using the system. Furthermore, I could see the problem arising of people starting a fuss about specific people applying the rubric unfairly. With no rubric, it's all personal opinion and everyone knows that.

Besides, rubricless rating systems still work well in other applications. For instance, on Amazon.com, you rate a product from 1 to 5 stars. No rubric is provided, but the reviews still serve a useful purpose. You select a number of stars, and read the reviews that gave that number of stars, and you figure out which reviews tend to scrutinize the item most fairly and logically. You may find that all the 1-star reviews are by people who didn't read the instructions, and thus you can ignore them. Alternately, you may find that the 5 star reviews are from people who don't know how to tell whether the product in question serves its purpose well compared to similar products, and you can ignore them.

I think the system Bain and I came up with for ignoring users you frequently disagree with or searching only for ratings by people you tend to agree with fits its purpose well in that regard. Even people who don't review or discuss don't pose a problem, because you can base whether you agree with that person on whether you agree with reviewers who rated the scenario similarly.
The unofficial Blades of Exile merchandise shop: No markups, cheap prices, quality printing! Spread the word!

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 23 July 2009 - 09:44 AM #59 Housekeeping: Revised Blades of Avernum Ratings System

So, back to the questions.

1 - Do we need to change how scenarios are rated? You know, the methodology.
2 - Does the CSR need moving? Certain answers to #1 would mandate it, others wouldn't.

And no, moving it because some folks can't post on SV is not a valid answer. Some folks can't post here. Actually, probably more BoX people are restricted from posting here than there.
Got pr0n? Join the Spiderwebbers!

Duck in a Top Hat Duck in a Top Hat

Shadowwalker

  • Member
    • Member ID: 6,265
  • 514 posts

Posted 23 July 2009 - 09:58 AM #60 Housekeeping: Revised Blades of Avernum Ratings System

1. I have heard some ideas that I think would be vastly better. I say we pick the best one, and change it.
2. Yes. No one should have to wade through porn to read a scenario rating.

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 23 July 2009 - 10:21 AM #61 Housekeeping: Revised Blades of Avernum Ratings System

I just checked, and couldn't find any porn in the current CSR. Could you PM me the link please?
Got pr0n? Join the Spiderwebbers!

Lazarus. Lazarus.

Hand of Avadon

  • Member
    • Member ID: 3,972
  • 1,223 posts

Posted 23 July 2009 - 10:51 AM #62 Housekeeping: Revised Blades of Avernum Ratings System

I'm going to agree with Salmon. If we're going to move it, it should be either because we're putting in some new rating system that can't be done using the current forum setup, or it should be moved here where it would be most widely accessible (And I'm not talking about forum preferences of a handful of users here, I'm talking about being accessible to all the users who don't frequent any satellite, and could care less about SV vs CRF.) Moving the CSR from one satellite to another satellite with a stricter CoC is just a waste of time. We keep our shenanigans out of the holy grounds of the CSR, so you can just drop the stupid "Think of the children!" routine.

As far as new rating systems go, there are a few major problems. Firstly, the work that's going to go into changing it. Since the new setups require fancy web scripting and such, this basically would all fall on the shoulders of one or two users (or just Aran.) Secondly, we have hundreds of old reviews that won't fit this new format, what do we do with those? Do we ask users to repost all their old scores in the new format? That would be a huge pain, and totally pointless. Some reviewers are no longer active members, and others are still active but don't spend enough time here anymore to be expected to move all their reviews. We could list the old reviews alongside the new reviews, just the way they are as a block of text. But that just comes off as half-assed to have a huge portion of the CSR be in one format and a few of the new ratings be in a different one. So do we throw all the old reviews away and start fresh? That's a pretty big waste since expecting a new CSR to catch up in size to the current one is probably not reasonable.

So it's not just a matter of "This looks like a better system, and we're jonesing for change, so let's do it!"
"It's all that weiner kid TM's fault." -Nikki

Enraged Slith Enraged Slith

Revenge of the Slith

  • Member
    • Member ID: 27
  • 2,634 posts

Posted 23 July 2009 - 11:04 AM #63 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Smiley-Faced Duck
2. Yes. No one should have to wade through porn to read a scenario rating.
How is that remotely relevant to where the CSR is now?

Kelandon Kelandon

!!!

  • Global Moderator
    • Member ID: 2,682
  • 9,786 posts

Posted 23 July 2009 - 11:10 AM #64 Housekeeping: Revised Blades of Avernum Ratings System

CSR on SV never made sense to me in the first place.

The final number for a CSR rating seems like crap to me. How do you rate a scenario with a single number? I would've played a hell of a lot more BoE scenarios if I could've ever found what I was looking for. My favorite scenario, The Election, was buried down in the 7-ish area, and I don't have any idea how I ever found it in there. Scenarios that I found deeply frustrating were all the way up in the 9's. CSR's numbers concealed more than they revealed, as far as I was concerned.

How do you compare Nobody's Heroes, which is a short, humor-based scenario with basically no combat, to Exodus, which is a very long, serious scenario with tons and tons of combat, if you're using just a single number? How do you compare Nine Variations on Point B, which is a meta-scenario port, to the High Level Party Maker, which is a utility scenario, with just a single number? The number is useless — worse than useless, because it directs attention away from the substantive issues in the scenario.

What we need is a set of tags (number of towns, number of outdoor sections, etc.), some of which are subjective (degree of polish), ideally in a searchable format. If I want to pull up a short, humor-based scenario that is well-polished, with or without combat, I should be able to do that quickly and easily. Right now, I can't.

Feedback for designers can take some other completely different form. But this is what the players need.
VCH: I believe we settled this way back when: Kelandon was the most attractive.
SoT: You'd be happy, too, if you were such a clever spider.
Ephesos: In conclusion, yarr.

Kelandon's Pink and Pretty Page!!

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 23 July 2009 - 11:35 AM #65 Housekeeping: Revised Blades of Avernum Ratings System

Kel has a legitimate beef here, imo, and is really talking about the only reason for making any change. After all, the purpose of this whole discussion is to increase the audience of Blades.

Now, most of us who review already break it out into various subcomponents, giving our x-10 rating for each, and then combining and using some sort of averaging (see Nioca's in SV for reference, no link provided). If there was a way of making that a standard, then that would be one system. Instead of one number, there could be 4 (plot, combat, etc etc).

Another would be to break out the existing single score ranking system into multiple grouplings, based on those tags. A scenario might qualify under two or more groups as well. So, a Combat group, Large group, Small group, Plot group. The designer could specify, or in the case of older BoE, a resident B(ain)oEophile could do it for us.
Got pr0n? Join the Spiderwebbers!

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 23 July 2009 - 12:37 PM #66 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Naughty Salmon
So, back to the questions.

1 - Do we need to change how scenarios are rated? You know, the methodology.
2 - Does the CSR need moving? Certain answers to #1 would mandate it, others wouldn't.

And no, moving it because some folks can't post on SV is not a valid answer. Some folks can't post here. Actually, probably more BoX people are restricted from posting here than there.

Exactly. There are some people who can't post everywhere, for whatever reason. So why not move it to a place where differnces don't matter?

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,402 posts
  • LocationMassachusetts

Posted 23 July 2009 - 01:17 PM #67 Housekeeping: Revised Blades of Avernum Ratings System

Salmon, when you say "Some folks can't post here" I assume you're referring to TM and TGM. Other members of the community are more than happy to channel them for legitimate business and the administration seems okay with that, so I don't see that as too much of a problem. SW is probably better than SV if we keep it message-board based.

Anyway, I've modified my Blades Forge-based idea yet again in light of recent discussion, so here it is:

---

There would be three specific Quality rating categories on a scale of 1-10, with 1 being absolutely bereft of any sort of redeeming quality whatsoever and 10 being absolutely perfect, all listed as numbers only except 1 and 10. For each of these ratings there would also be "Not Applicable" and "Abstain/No Opinion" options. There would also be a similarly-scaled Overall Quality rating, except it would be to one decimal place, like CSR.

There would also be three Genre scales. These would have five (or whatever odd number) descriptive ratings with an clearly defined center, that serve to indicate what sort of scenario it is.

All ratings would be optional except (possibly) Overall Quality, which might be required for all rating submissions. A user may modify his past ratings at will.

The above describe Rating system will be placed at the top of a scenario's Discussion Thread for reviews and other discussion, sort of like a poll on a typical message board (only better). Results would have to be visible by default, unlike message board polls, because first you look at the rating to decide what to play, then you play it, and only then do you actually rate it. Next to the results will be a button for "Ratings By User" that will display the ratings by user and allow you to uncheck or recheck boxes next to each user, in order to exclude or include a specific user's ratings in the calculated, searchable averages.

Below the Ratings would be a Discussion Thread for textual reviews and discussions. Spoiler tags MUST be enforced, but spoilers are allowed. You can discuss the player-end aspects of the scenario, the designer-end aspects, or both; it doesn't matter. The actual rating aspect of the system is strictly player-oriented, however. This is both so that players won't have to deal with scenarios being rated highly for things that don't affect play, and to prevent people who don't know what they're talking about from bringing down advanced scenarios. (Actually, perhaps there could be an additional scale that's only available if you've released a scenario. That can be discussed later, though.)

Lastly, as is already present on the Blades Forge, you can Tag the scenario with keywords that indicate noteworthy aspects of the scenario.

---

The [b[Overall Quality Rating[/b] is independent, but has an optional "Calculate Average" button.

Each of the three specific categories would also have an explanation:

1. Writing
This category encompasses storyline, plot, themes, dialogue, descriptions, cliché or lack thereof, etc. The literary aspects of the scenario.

2. Gameplay
This category encompasses combat, puzzles, balance (how well the combat, treasure, EXP, etc. interact), engine modifications (such as the water system in "Destiny of the Spheres" or the rune system in "Nebulous Times Hence"), flow (or something, not sure what to call it. For instance, do you constantly have to blindly guess what you have to do or say to progress the scenario, or are the outdoors so unnecessarily large that it takes three weeks real time to get from one town to the next?), etc. What the player actually does while playing the scenario.

3. Aesthetics, Functionality, Etc.
This category includes graphics, spelling and grammar, town design, cutscenes, technical noding/scripting, bugginess, whether the scenario contains far too much gratuitous foul language/lewdness/etc. (if that's important to the person rating the scenario), artistic merit, etc. The presentation of the scenario, things whose primary purpose is to enhance (or detract from) immersion in the scenario. I know some people want a separate scale for graphics so we can exalt Ephesos's pretty trees, but ultimately I feel that people who are playing these games generally aren't doing it for the graphics, so I don't feel it's that important. If you want, you can always Tag the scenario with keywords like "beautiful" or "visually stunning" or whatever.

---

After those categories, the Genre Scales:

1. Serious<->Humorous +N/A (in case someone releases a scenario with nothing but combat and no text, which for all we know may happen, we'll need an N/A checkbox)

2. Hack-and-Slash Combat and No Puzzles<->Strategic Combat and/or Puzzle-Heavy +No Combat or Puzzles (I think strategic combat and puzzles should be together because really, strategic combat is a sort of puzzle. This scale will move right if there are puzzles, even if there is hack'n'slash or no combat, for instance, or if there's tactical combat but not puzzles, or both.)

3. Linear<->Open-Ended (Can anyone think of a scenario where this might need an N/A? Not necessarily a BoX "scenario" but scenario in the literal sense.)

---

I don't think we need a rating scale for short/long/epic, because the length of the scenario isn't subject to opinion. We could just include a Number of Towns and Size of Outdoors field in the database, I think. Same goes for recommended party level/prefab/singleton. We should be careful not to just use the level listed in the scenario, though, if the readme says otherwise. I don't know if this matters much in BoA, but in BoE, some scenarios were listed as Very High level just to prevent monster HP doubling. Lastly, for utility scenarios, the "genre" really isn't open to interpretation either. The fact that it is a utility scenario and perhaps a more specific categorical tag should be included in the database when the scenario is posted.

---

Of note is that the Overall Quality rating is the same scale as CSR's system, which makes it easy for CSR ratings to be imported into the Blades Forge (or whereever) system. The ratings would go into Overall Quality, and the reviews would go into the discussion thread. As with the original Blades Forge policy regarding old scenarios whose authors are no longer around, old CSR reviews will be placed under the control of the user who posted them when they register at the Blades Forge. This will of course be done manually, although the original porting from CSR to Blades Forge could probably be automated.

---

Any comments are welcome.
The unofficial Blades of Exile merchandise shop: No markups, cheap prices, quality printing! Spread the word!

Celtic Minstrel Celtic Minstrel

Eye of Avadon

  • Global Moderator
    • Member ID: 5,296
  • 3,868 posts

Posted 23 July 2009 - 02:47 PM #68 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: The Almighty Doer of Stuff
Besides, rubricless rating systems still work well in other applications. For instance, on Amazon.com, you rate a product from 1 to 5 stars. No rubric is provided, but the reviews still serve a useful purpose. You select a number of stars, and read the reviews that gave that number of stars, and you figure out which reviews tend to scrutinize the item most fairly and logically. You may find that all the 1-star reviews are by people who didn't read the instructions, and thus you can ignore them. Alternately, you may find that the 5 star reviews are from people who don't know how to tell whether the product in question serves its purpose well compared to similar products, and you can ignore them.
The Blades Forge already has a rating system just like this, though I'm not entirely sure if it's available on the scenario pages.

Originally Posted By: Naughty Salmon
Another would be to break out the existing single score ranking system into multiple grouplings, based on those tags. A scenario might qualify under two or more groups as well. So, a Combat group, Large group, Small group, Plot group. The designer could specify, or in the case of older BoE, a resident B(ain)oEophile could do it for us.
Again, this is possible on the Blades Forge by using labels. There's no limit to what kind of labels can be applied to a scenario.



Of course, with some programming ADoS's idea above could also be implemented at the Blades Forge. The discussion thread already exists in the form of comments, doesn't it? Or if not it shouldn't be too hard to add since it's implemented elsewhere. Alternatively it could be placed in the Blades Forge forum with a link to the scenario page.
Exile 2 Book of Items (I didn't create it, I just converted it to HTML.)

"Man, I know how you feel. I once spent an hour playing WordPad before realising that it was a text editor." – Thuryl

"Dikiyoba just hopes no one ever blows up Saturn. Getting those rings back into their proper position would take hours."

"—Alorael, who spells phoenetically. No matter how much his orthography is a wreck, intelligibility rises from the ashes."

"I've never played a Spiderweb game. I didn't even know SW made games until I saw this thread." —Nikki

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 23 July 2009 - 03:22 PM #69 Housekeeping: Revised Blades of Avernum Ratings System

This is also readily available (responding to Crazy Minstrel) at the SV within the current tables. Just break the master list into multiple lists. This can be done manually by anyone with mod access to that forum.
Got pr0n? Join the Spiderwebbers!

Duck in a Top Hat Duck in a Top Hat

Shadowwalker

  • Member
    • Member ID: 6,265
  • 514 posts

Posted 23 July 2009 - 05:43 PM #70 Housekeeping: Revised Blades of Avernum Ratings System

Quote:
I just checked, and couldn't find any porn in the current CSR. Could you PM me the link please?
While there is nothing of that nature in either of the CSR forums I really just hate the idea of even having to go to that site when I know it really is in the other forums.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users