Jump to content


Housekeeping: Revised Blades of Avernum Ratings System


  • Please log in to reply
385 replies to this topic
The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,415 posts
  • LocationMassachusetts

Posted 21 July 2009 - 07:35 AM #1 Housekeeping: Revised Blades of Avernum Ratings System

http://forums.calref.co.cc/index.php?board=39.0

Many people have cited issues they have with the current Comprehensive Scenario Rankings system we have over at Shadow Vale. Some have argued for creating an entirely new system for rating and reviewing scenarios. Others say we should simply improve the current CSR to correct its failings. What do you think we should do? Have your say.

Voting has begun on the direction we should take the new (or simply modified) system, here. The poll will be restarted as new ideas are proposed, until the suggestions stop and we have a clear winner.

Drakefyre Drakefyre

Manifest Destiny

  • Member
    • Member ID: 8
  • 9,598 posts

Posted 21 July 2009 - 08:20 AM #2 Housekeeping: Revised Blades of Avernum Ratings System

What's wrong with CSR?
“Iron rusts from disuse; water loses its purity from stagnation ... even so does inaction sap the vigour of the mind.” - Leonardo da Vinci
====
Drakefyre's Demesne  - Happy Happy Joy Joy
Encyclopedia Ermariana  - Trapped in the Closet
====
You can take my Mac when you pry my cold, dead fingers off the mouse!

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 21 July 2009 - 08:56 AM #3 Housekeeping: Revised Blades of Avernum Ratings System

Quote:
What's wrong with the CSR?

A couple of things. For one, its the reviews themselves. Most of them tend to be unhelpful to the designer or even the player (who is arguably the lesser important of the two and should therefore receive less attention). Commonly, a large amount of reviewers (including me, in all honesty)tend to be short, meaningless posts with a score. Reviewers would point out how amazingly technically advanced scenario's are (Blades of Rogue particularly) and hand it a score fractions over a 2. Most of the recent reviews have been more thoughtful, true, but its unlikely this trend will continue (I admit this is more of a personal opinion then fact). A fresh atmosphere is a good way to encourage people to "begin anew".

The second reason is that the current review system is too vague. Some reviewers start with 5's because they seem like the "fairest" number. Others start with 6 to take into account giving people "the benefit of the doubt", or some such like that. Other people (such as myself) just jamble all of the scenario in the review and try to pull a score out of their ass. Others have a fixed ratio and system (Lenar Lambs) and use that to decide a score. And the scores themselves aren't consistent. Some people consider a 5 to the the "average" like a C+ or something, while others (like myself) consider 7 to be the "average" score. There are plenty, plenty of other people that think differently.

As you can see, its really a mess. We need a fixed way to review a scenario that is both helpful to the player and the designer alike.

The Great Archon

Enraged Slith Enraged Slith

Revenge of the Slith

  • Member
    • Member ID: 27
  • 2,634 posts

Posted 21 July 2009 - 09:00 AM #4 Housekeeping: Revised Blades of Avernum Ratings System

Some people want to enforce standard reviews so they can judge the merit of someone's opinion, rather than just accept a blind score.  There is also some concern about shock imagery, which doesn't actually occur in the associated boards.

Eclipse Eclipse

Watcher

  • Member
    • Member ID: 3,901
  • 171 posts

Posted 21 July 2009 - 09:06 AM #5 Housekeeping: Revised Blades of Avernum Ratings System

How about discarding the point system altogether? How about just having it set up like an oreo.

Summary of things you liked. (1 paragraph)

Summary of things you didn't like. (1 paragraph)

Do you reccomend? If not, why? (1 paragraph)

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 21 July 2009 - 09:08 AM #6 Housekeeping: Revised Blades of Avernum Ratings System

That's the ideal situation, where we don't have to rely on a damn number, but people in general want to know right off the bat if something is good or not, including myself. So a number, or at least some official way of classifying it, is pretty much a necessity.

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,415 posts
  • LocationMassachusetts

Posted 21 July 2009 - 09:14 AM #7 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Master Ackrovan
A couple of things. For one, its the reviews themselves. Most of them tend to be unhelpful to the designer or even the player (who is arguably the lesser important of the two and should therefore receive less attention). Commonly, a large amount of reviewers (including me, in all honesty)tend to be short, meaningless posts with a score. Reviewers would point out how amazingly technically advanced scenario's are (Blades of Rogue particularly) and hand it a score fractions over a 2. Most of the recent reviews have been more thoughtful, true, but its unlikely this trend will continue (I admit this is more of a personal opinion then fact). A fresh atmosphere is a good way to encourage people to "begin anew".

I'd say the player is more important than the designer as far as reviews go, although there needs to be a good way for designers to get feedback as well. I'd also say scenarios that are technically advanced but not really fun to play should be getting low scores, although with BoR, from what I understand, it was getting very high scores for being complicated technically, despite not being particularly interesting from a player's perspective. I definitely agree with the comment about the fresh atmosphere, though, and the second paragraph is accurate.

We're building off threads on Shadow Vale about the shortcomings of the CSR, by the way. Here they are:
http://forum.nethergate.net/index.php?showtopic=1477
http://forum.nethergate.net/index.php?showtopic=1504

EDIT: As for why we need numbers, nobody wants to read through 400 threads to find the scenarios they might like. We need some way of searching more efficiently.

sylae sylae

Wisebottom

  • Global Moderator
    • Member ID: 5,482
  • 3,336 posts
  • LocationPortland, Oregon, United States

Posted 21 July 2009 - 09:37 AM #8 Housekeeping: Revised Blades of Avernum Ratings System

Okay, regarding the vote kindly linked to by ADoS in the first post, So as to maintain security and prevent cheating from guest voting, registration is required to vote. you may do so by going here and filling out the form. You don't even have to do a stupid e-mail confirmation thing! (Also, for you paranoid types, you have an option to hide your mail address from registered members as well as guests).

Thanks,
w-dueck

Your local CRF admin type.

Janitorial Closet Janitorial Closet

Mercenary

  • Member
    • Member ID: 5,444
  • 94 posts

Posted 21 July 2009 - 10:10 AM #9 Housekeeping: Revised Blades of Avernum Ratings System

I really have no reason to post at the CRF, and only registered to vote. If the account is deleted due to not posting (which I want to happen) is the vote devalidated?
"why would some one in their right mind pee/poop on the geneforge?"- Safey

"a spam topic were people just post random things to raise their thing"- T.E.N.Bob

Dintiradan Dintiradan

Eye of Argon

  • Member
    • Member ID: 4,273
  • 3,697 posts

Posted 21 July 2009 - 10:23 AM #10 Housekeeping: Revised Blades of Avernum Ratings System

I may make a longer post when I get home, but here's two sentences for you all now:

The CSR is used only to benefit the players, not the developers.
Reviewers send an e-mail to the designer, talking about the scenario in depth.

Discuss.
^<@<: Pronounced exactly as it's spelt.
Design Tutorial (incomplete) | The Metagame | AIMhack Crack Pairings
Artemis (Planning Thread)
Last ten games. What have you been playing lately?

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 21 July 2009 - 11:21 AM #11 Housekeeping: Revised Blades of Avernum Ratings System

:facepalm:

How is having a discussion of this type on three different forums helpful?
Got pr0n? Join the Spiderwebbers!

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,415 posts
  • LocationMassachusetts

Posted 21 July 2009 - 11:29 AM #12 Housekeeping: Revised Blades of Avernum Ratings System

I was hoping to have all discussion on CRF. I was just posting a heads-up on SV and SW, but apparently Drakey and Dinti don't want it that way. *shakes finger*

Let's redirect all discussion to the CRF, please.

Niemand Niemand

Hand of Avadon

  • Moderator
    • Member ID: 3,580
  • 2,138 posts
  • LocationWisconsin

Posted 21 July 2009 - 12:05 PM #13 Housekeeping: Revised Blades of Avernum Ratings System

You know, it really would make the most sense to either continue the discussion where it was (SV) or just hold it where the most people are (here).

Originally Posted By: Dintiradan
Reviewers send an e-mail to the designer, talking about the scenario in depth.
This is an interesting suggestion, but it would prevent designers from learning from the results of other designers' scenarios, unless designers made an effort to summarize and publish the feedback they received. I'm not saying that designers necessarily do get a lot of mileage out of the reviews they see of others' work, but it is something that the current system provides.
Possibly you suppose that [he] is a harmless fellow, who with vacant mind recites some empty formula,
traces a wandering sign in the air, and with palsied hand shakes a wand the while he gibbers his insanity at the yawning moon.

Only Game Only Game

Alorael

  • Administrator
    • Member ID: 276
  • 22,118 posts
  • LocationElsewhere

Posted 21 July 2009 - 12:18 PM #14 Housekeeping: Revised Blades of Avernum Ratings System

Why not have two part reviews? The score could be based on the first part, which is entirely player perspective and all about fun and enjoyment. The second part could dig into the technical details and comment on scenario design. If you were really ambitious the second part could even have its own entirely different score!

—Alorael, whose feelings about reviews are that the numbers are usually unhelpful. The commentary is always where you can find the useful information. More text is always better.

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 21 July 2009 - 12:33 PM #15 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: The Almighty Doer of Stuff
Let's redirect all discussion to the CRF, please.


That's not going to happen, I'm afraid. frown
Got pr0n? Join the Spiderwebbers!

VCH VCH

Eye of Avadon

  • Member
    • Member ID: 210
  • 3,490 posts

Posted 21 July 2009 - 12:45 PM #16 Housekeeping: Revised Blades of Avernum Ratings System

The discussion should obviously be here. SV and CRF are minor satellites. I would assume that everyone that has an interest in BOA comes here at least several times a month. A sticky should be made in the BOA section of the SW forums for the discussion. If we continue to host the discussion were only established and well known people will post then the result, I believe, will be at best neutral.
Мама, я в России!

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 21 July 2009 - 12:53 PM #17 Housekeeping: Revised Blades of Avernum Ratings System

Quote:
You know, it really would make the most sense to either continue the discussion where it was (SV).....)\.

I'm sorry to say that that is not possible for me frown

tridash tridash

Calculated.

  • Member
    • Member ID: 2,261
  • 299 posts
  • Locationuk

Posted 21 July 2009 - 01:02 PM #18 Housekeeping: Revised Blades of Avernum Ratings System

I think this is probably the best place to continue the discussion, since it's the most neutral of the suitable sites.

Some ideas posed as questions that may or may not be useful:

-Would Aran be willing for anyone else to work on the Blades Forge? I understand that he's busy, but if the CSR has to go somewhere else, that would seem a good place for it to go.
-If the CSR is moved, might it be worth registering a separate domain name for it so that people don't object to whichever domain it is that it's on?
-Would a chat be useful to bash out some more developed ideas that could then be voted on or discussed later?

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 21 July 2009 - 01:12 PM #19 Housekeeping: Revised Blades of Avernum Ratings System

Quote:
-Would Aran be willing for anyone else to work on the Blades Forge? I understand that he's busy, but if the CSR has to go somewhere else, that would seem a good place for it to go.

This was actually discussed in a chatroom. ADoS sent Arancaytar an Email about it. He hasn't gotten back to him yet.

Quote:
-If the CSR is moved, might it be worth registering a separate domain name for it so that people don't object to whichever domain it is that it's on?
Yes indeed

Quote:
-Would a chat be useful to bash out some more developed ideas that could then be voted on or discussed later?

There is CRF's chat, but that is of course not a good idea. If we all want to do this in a Chatroom, AIM would be the best bet. And the fairest.

Thoughts?

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,415 posts
  • LocationMassachusetts

Posted 21 July 2009 - 01:15 PM #20 Housekeeping: Revised Blades of Avernum Ratings System

Plus if we use CRF's chat, we can't have more than 8 people before the server crashes. >_<

Duck in a Top Hat Duck in a Top Hat

Shadowwalker

  • Member
    • Member ID: 6,265
  • 514 posts

Posted 21 July 2009 - 01:22 PM #21 Housekeeping: Revised Blades of Avernum Ratings System

I'm up for a chat on AIM. I've got some ideas, but I doubt they're any good.

BainIhrno BainIhrno

Hand of Avadon

  • Moderator
    • Member ID: 3,311
  • 1,312 posts

Posted 21 July 2009 - 02:45 PM #22 Housekeeping: Revised Blades of Avernum Ratings System

I'd like an AIM chat as well, this whole thing is sorta confusing me.. the problem I have with the current CSR, is the ratings seem too specific. To me, I'm feeling, is there really much difference between a 7.5 and a 7.6. I have no problem with a number scale, but think there should be fewer rate options.

Enraged Slith Enraged Slith

Revenge of the Slith

  • Member
    • Member ID: 27
  • 2,634 posts

Posted 21 July 2009 - 02:57 PM #23 Housekeeping: Revised Blades of Avernum Ratings System

Hurr hurr, we can't even agree on where to hold the discussion, much less what the problem is.  Did people even have these discussions back during BoE's prime?  If not, what is causing all the hubbub now?

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 21 July 2009 - 03:49 PM #24 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Slith
If not, what is causing all the hubbub now?
1. People are unfamiliar with the formalities of rating a scenario.
2. There is a certain positive pressure to rate scenarios.
3. The current CSR location is problematic for some people.
4. There is an effort to gain participation in the BoX community, and many current community members see that the CSR in place is cumbersome and may, in fact, be an obstacle for new players.
5. ...
6. Profit.

Got pr0n? Join the Spiderwebbers!

Arancaytar Arancaytar

Archivist

  • Global Moderator
    • Member ID: 2,001
  • 10,474 posts
  • LocationThe Empire of Ermarian

Posted 21 July 2009 - 04:20 PM #25 Housekeeping: Revised Blades of Avernum Ratings System

All this satellite bickering is a load of bollocks.

This balkanization, more than anything else, is the cause Blades has no healthy community anymore. And because the value of Blades is user-added, the lack of a healthy community means Blades never sold as well as it could have, which in turn made JV abandon the concept.

Now I'm seeing people compare this conflict to other historical conflicts, while others respond that this conflict is by far not as cool as the conflicts we used to get back in our day.

Whether Calamity Refuge versus Shadow Vale is like Polaris versus Desperance is not the point. The point is that Polaris versus Desperance was a broken concept. Polaris and Desperance are dead. They are dead like the Chance Forums are dead - the only difference is the nostalgia, and communities based on nostalgia don't work. If you want to live long as a satellite, you need to avoid these mistakes, not re-enact them.

...

Now that I've got this off my chest, I'll gladly participate in this discussion at any place you'll name. If we can agree on a scoring system, maybe it can even be integrated with the Blades Forge's scenario list.

Speaking of which, as I said to ADoS privately, I would very much like the next Blades Forge to become a team project, because I'm not up to doing that on my own again (witness that the Forge hasn't really been overhauled in almost two years). Anyone who messes about with PHP and Drupal would be welcome. smile

Ephesos Ephesos

A God in Mote's Eye

  • Moderator
    • Member ID: 2,749
  • 6,157 posts

Posted 21 July 2009 - 06:05 PM #26 Housekeeping: Revised Blades of Avernum Ratings System

My thoughts have largely been stated by other members.

Originally Posted By: Naughty Salmon
How is having a discussion of this type on three different forums helpful?

It isn't, duh.

Originally Posted By: VCH
The discussion should obviously be here. SV and CRF are minor satellites.

Wholeheartedly agreed. I feel like moving this thread.

Originally Posted By: Naughty Salmon

6. Profit.

Hear hear!

Originally Posted By: Arancaytrus
All this satellite bickering is a load of bollocks.


Aran wins at life. And this thread is moving. Now.

The Almighty Doer of Stuff The Almighty Doer of Stuff

Heart of Avadon

  • Member
    • Member ID: 72
  • 4,415 posts
  • LocationMassachusetts

Posted 21 July 2009 - 06:32 PM #27 Housekeeping: Revised Blades of Avernum Ratings System

The reason I put it in General was because I wanted it to be about BoE as well as BoA, but I guess it doesn't matter much. At any rate, Spidweb is the best place for the discussion, definitely.

Lilith Lilith

in before the apocalypse

  • Global Moderator
    • Member ID: 669
  • 19,845 posts
  • LocationMelbourne, Australia

Posted 21 July 2009 - 06:51 PM #28 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Enraged Slith
Hurr hurr, we can't even agree on where to hold the discussion, much less what the problem is.  Did people even have these discussions back during BoE's prime?  If not, what is causing all the hubbub now?


Oh boy, did they ever. The word "rubric" was a running joke in the BoE community ever since a certain member made a ridiculously overcomplicated and lopsided checklist for giving every scenario an objective score out of 71. It was a bad idea then and it's a bad idea now.

Acky Acky

Hand of Avadon

  • Member
    • Member ID: 5,375
  • 2,427 posts
  • LocationPort Charles, Florida

Posted 21 July 2009 - 07:00 PM #29 Housekeeping: Revised Blades of Avernum Ratings System

Yikes. But no rubric is just as bad, as I mentioned earlier. There needs to be some sort of standard way that a scenario is reviewed that's easy to follow and provides enough room to be helpful to both the player and the designer. But having nothing may be as bad as having something bad.

Niemand Niemand

Hand of Avadon

  • Moderator
    • Member ID: 3,580
  • 2,138 posts
  • LocationWisconsin

Posted 21 July 2009 - 07:07 PM #30 Housekeeping: Revised Blades of Avernum Ratings System

Originally Posted By: Arancaytar
Speaking of which, as I said to ADoS privately, I would very much like the next Blades Forge to become a team project, because I'm not up to doing that on my own again (witness that the Forge hasn't really been overhauled in almost two years). Anyone who messes about with PHP and Drupal would be welcome.
While I lack any particular skill to help, I would be excited to see that happen. (Although I stand by my offer to work on a desktop client to interface with part or all of the Forge.) One thing that I would personally be curious to learn more about is what the necessary work on the Forge actually entails; basically, what the project's development checklist looks like.
Possibly you suppose that [he] is a harmless fellow, who with vacant mind recites some empty formula,
traces a wandering sign in the air, and with palsied hand shakes a wand the while he gibbers his insanity at the yawning moon.

Lazarus. Lazarus.

Hand of Avadon

  • Member
    • Member ID: 3,972
  • 1,223 posts

Posted 21 July 2009 - 08:04 PM #31 Housekeeping: Revised Blades of Avernum Ratings System

The CSR's failing isn't the scoring system, or whether rubrics are used. The problem is that nobody bothers to write insightful reviews, and I don't think moving sites or adding categorical scores are going to solve that. The CSR has been the way it's been for years; it works when there's a healthy community of designers and players writing good reviews (see: geocities lyceum and most of ezboard Lyceum) and it sucks when the community is mostly disinterested (see: SV, or even worse the end of ezboard Lyceum.) I'm not convinced that any amount of rubrics will fix the CSR unless you fix Blades first, and good freaking luck with that.

While it would be cool to have ratings somewhere like the Blades Forge, I don't expect that to really fix anything and I couldn't demand any more of Aran than he's already done.
"It's all that weiner kid TM's fault." -Nikki

Schoolin' Salmon Schoolin' Salmon

½ Man, ½ Amazing

  • Member
    • Member ID: 3,699
  • 5,572 posts

Posted 21 July 2009 - 09:31 PM #32 Housekeeping: Revised Blades of Avernum Ratings System

We don't need a magical scoring system. We need good reviewers. We need lucid individuals who understand the capacity of Blades, the capabilities of the designer, the current level of innovation. We need communicators who will take the time to READ the scenario documentation and learn the author's purpose in design. They can then write a paragraph, and make an informed scoring decision on a scale of 0-10.
This can, and has worked. Unfortunately, it means some people shouldn't be reviewers. That is a harsh thing to have to tell someone, and this type of community, there is no one in a position to do it.
Got pr0n? Join the Spiderwebbers!

Enraged Slith Enraged Slith

Revenge of the Slith

  • Member
    • Member ID: 27
  • 2,634 posts

Posted 21 July 2009 - 09:43 PM #33 Housekeeping: Revised Blades of Avernum Ratings System

If we had the web designer, we could honestly just set up a website with all the scenarios in a list with a little 5 star visual and people could just put the number of stars they thought a scenario earned as well as a comment section (a la Youtube).  Visuals are easy to conceptualize and really cut into the heart of what our current system is.  This would help players choose what they wanted to play, which, I think, is the most important aspect of the CSR.  Most may want in-depth analyses, but those don't help anyone but the designer.

Designers should really look to beta testers for help and comments or ask specific questions.  I know Nikki was more than happy to comment on my work in progress, so we should just pawn everything off on him.

Also, I'm pretty much in agreement with Laz (who needs to make another awesome scenario).

VCH VCH

Eye of Avadon

  • Member
    • Member ID: 210
  • 3,490 posts

Posted 22 July 2009 - 01:42 AM #34 Housekeeping: Revised Blades of Avernum Ratings System

Do you think it would be possible to institute a letter grade scheme for ranking scenarios? We could have descriptions of each letter grade e.g. an A+ is an outstanding scenario that shows exceptional plot, combat etc., a B is . . .
Мама, я в России!

Drakefyre Drakefyre

Manifest Destiny

  • Member
    • Member ID: 8
  • 9,598 posts

Posted 22 July 2009 - 05:35 AM #35 Housekeeping: Revised Blades of Avernum Ratings System

I think that numbers are fine. It doesn't matter if people have different scales as long as their scales are internally consistent and they rate enough scenarios. The numbers provide a basis for what scenarios people read about - they choose the threads based on the scores, then choose scenarios based on the comments in the threads.

Wisdom of the crowds and all.

Move it to a "neutral" place, sure. But putting more rules/restrictions on reviewing isn't what is needed. Score and comments is the simplest effective method for reviewing scenarios.

DON'T provide any guidance for what a score means. Everyone has their own ideas of what a "9" scenario is. The point of Community (sure Comprehensive works too) reviews is that everyone's views are averaged into a sense of what scenarios are best.
“Iron rusts from disuse; water loses its purity from stagnation ... even so does inaction sap the vigour of the mind.” - Leonardo da Vinci
====
Drakefyre's Demesne  - Happy Happy Joy Joy
Encyclopedia Ermariana  - Trapped in the Closet
====
You can take my Mac when you pry my cold, dead fingers off the mouse!




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users