Jump to content

Housekeeping: Revised Blades of Avernum Ratings System


Recommended Posts

  • Replies 385
  • Created
  • Last Reply

Top Posters In This Topic

Nioca, I wasn't implying that some elite should have control over reviews. Instead, just the simple forum-header instructions would read "If you wish to post a review, PM a mod with the review. Once that review has been approved (then cite comprehensive list of reasons for disapproval) you will be able to post freely within this sub-forum, knowing that such permission is contingent on compliance with the rules."

 

As far as visibility goes, the link to the CSR is at the top of the Blades forum. It can't get more visible than that, can it?

 

stareye - One of Kel's suggestions seems to have been left to the side, although Nioca did pick up on it a little. What would be helpful, as a player, would be to have the master list of reviewed scenarios be divided into smaller lists depending on scenario type. Now, I don't know what these types should be, but it would be a lot easier for someone only interested in puzzle scenarios to find their match if there was a separate listing of just that type.

As far as the extra trouble goes for separately approving each post, I disagree with the amount of control that policy gives to the moderators. There has to be some level of trust somewhere, someplace. It would be difficult to give an independent and fair review of a scenario created by one of the moderators, just as it would be difficult for a moderator/designer to post a unfavorable review. Besides, that hasn't been a problem with the reviews that exist.

As far as calling it quality control, that strikes of the elitism which some of our younger members are fighting against.

 

Sorry for being so rambly here, RL called me away several times over the past couple hours.

Link to comment
Share on other sites

Originally Posted By: Naughty Salmon
Nioca, I wasn't implying that some elite should have control over reviews. Instead, just the simple forum-header instructions would read "If you wish to post a review, PM a mod with the review. Once that review has been approved (then cite comprehensive list of reasons for disapproval) you will be able to post freely within this sub-forum, knowing that such permission is contingent on compliance with the rules."

Oh. Well in that case, we agree completely, and your idea is better than mine.
Link to comment
Share on other sites

I would like to reassert an idea from the Shadow Vale discussion. In addition to whatever style of numbers ranking ends up being used, pick 5 or 10 prolific reviewers, write a 2-3 sentence summary of what they like and dislike in scenarios, and list their top 5 or even top 10 favourite scenarios for each one. This allows players to look for somebody who has similar taste in scenarios, and see what that person liked.

 

IMHO, this is the best way to connect new players to scenarios they will enjoy quickly and without hassle. And IMHO, that's easily the best way for CSR to "expand the player base."

Link to comment
Share on other sites

What exactly is broken about my proposal, aside from it not being possible to host it on this message board? If we want it to be visible, we could put a link in the forum description and a visible, descriptive one in the forum header. Right now, in the BoE forum, the only link to Shadow Vale simply says "Shadow Vale" in the middle of a pile of other links. If it said in big letters, "Go here to read comprehensive scenario reviews and find a scenario you would like, or post your own review!" both in the forum description and in the forum header, that would probably attract plenty of attention.

Link to comment
Share on other sites

ADoS - It seems overwhelmingly complicated and ruley.

 

Other than visibility of the current CSR in the BoE forums, how do you see the CSR as failing to attract new players to the community? What aspect is hindering community expansion?

 

I have nothing against your system or forge, or whatever, but they still have to positively influence community expansion in order to be viable changes. Otherwise, it is so much busywork.

Link to comment
Share on other sites

It's not as complicated and ruley as it seems. That post was as long as it was because I had questions and explanations mixed in that aren't in the final product. I made a quick mockup of what it might look like to the user (hopefully a bit prettier):

http://ados.ermarian.net/BoX/ADoSRate.gif

 

As you can see, I rewrote a couple things, got rid of ambiguous terms, and moved "artistic merit" to "Creative Content". Above all the stuff in that image would be the actual results, which would be very similar-looking to the submission form, only it would have averages instead of choices (perhaps also showing what percent abstained or said "N/A"), and would have the "Ratings By User" button, and would lack the category descriptions.

 

EDIT: Oh, and I forgot, the specific ratings are integers. I don't feel like remaking the image, but you get the idea.

 

EDIT 2: Aran suggested sliders might be better than dropdown boxes for everything. The actual tool used can be decided later, I guess, but it gives the idea that it's not too complicated, I think.

Link to comment
Share on other sites

Quote:
One of Kel's suggestions...What would be helpful, as a player, would be to have the master list of reviewed scenarios be divided into smaller lists depending on scenario type.


That's fine. Categorizing is fine. The issue is all categorizing must be hard in the sense that people should not be able to disagree. Otherwise, it just becomes a rubric.

Quote:
As far as calling it quality control, that strikes of the elitism which some of our younger members are fighting against.


I'm calling it what it is, quality control. Standards will not be that high. It's just to keep things organized. One person, one review with one potential author response. One disadvantage to the Lyceum CSR was that long and heated discussions had to be cleaned up. I want to avoid that.

Quote:
What exactly is broken about my proposal, aside from it not being possible to host it on this message board?


That is exactly what is wrong. Let's keep everything centralized.
Link to comment
Share on other sites

Originally Posted By: *i
That's fine. Categorizing is fine. The issue is all categorizing must be hard in the sense that people should not be able to disagree. Otherwise, it just becomes a rubric.


Good. Let's have the scenario authors state where it belongs. Small or Large? Puzzle-type? Combat-Strategy type? Narrative-type? I would think that a scenario could be tagged into more than one category.

Originally Posted By: *i
I'm calling it what it is, quality control. Standards will not be that high. It's just to keep things organized. One person, one review with one potential author response. One disadvantage to the Lyceum CSR was that long and heated discussions had to be cleaned up. I want to avoid that.


I don't blame you. But, it hasn't been a recent problem, at least on SV. Why not start with the more open system (but not completely open) that Nioca and I are espousing, and if necessary you can restrict it from there. I know from experience that these things rarely become less restrictive.
Link to comment
Share on other sites

Originally Posted By: Naughty Salmon
Good. Let's have the scenario authors state where it belongs. Small or Large? Puzzle-type? Combat-Strategy type? Narrative-type? I would think that a scenario could be tagged into more than one category.
Salmon, you're a genius!

You've stumbled across the perfect reviewing system. Clearly, the biggest problem with reviewing systems are the reviewers themselves. Having scenario designers tag their own scenarios is a clear and logical step forward. But why stop there? Why not let the designers choose their own scores as well?

It's foolproof!
Link to comment
Share on other sites

I agree with Excalibur in that this should be put to a vote.

 

Quote:
'k, with regards to the genre-tagging issue. We could just add an extra thread, "Scenarios by Genre." Boom. Problem solves. Sticky it and we can move on.

 

Indeed.

 

As for the idea's, I'm a little torn, but I'll have to agree with *i's idea and move the CSR to here. Expanding the player base is certainly the most important thing we should aim for.

Link to comment
Share on other sites

Touché, fish-man, touché.

 

Excalibur: Hey, I remember reading a (somewhat) similar comment before. That is to say, voting won't count for anything unless the people willing to actually make the new CSR (at this point, just *i and the incumbent Tyranicus) want one.

 

(Link is NSFW - Shadow Vale. ;-) )

 

(No, really, posts preceding and following mine don't obey Spiderweb's CoC.)

Link to comment
Share on other sites

Originally Posted By: The Impossible and I
I would like to reassert an idea from the Shadow Vale discussion. In addition to whatever style of numbers ranking ends up being used, pick 5 or 10 prolific reviewers, write a 2-3 sentence summary of what they like and dislike in scenarios, and list their top 5 or even top 10 favourite scenarios for each one. This allows players to look for somebody who has similar taste in scenarios, and see what that person liked.

 

IMHO, this is the best way to connect new players to scenarios they will enjoy quickly and without hassle. And IMHO, that's easily the best way for CSR to "expand the player base."

 

This. Like a video store's employee picks. Back when people went to video stores before they knew what movie they were renting.

Link to comment
Share on other sites

Quote:
I don't blame you. But, it hasn't been a recent problem, at least on SV. Why not start with the more open system (but not completely open) that Nioca and I are espousing, and if necessary you can restrict it from there. I know from experience that these things rarely become less restrictive.


It is a lot easier to make things less restrictive than to make them more so later. Generally, people dislike it when privileges are taken away. I would say we should start more restrictive and then revisit the question later.

I have no problems with authors categorizing their scenarios. The caveat is that this must be known and authors can be punished in their reviews if they misclassify something. This puts a check on this.

I still contend that here is the best place. We have the most traffic and always will. Further, when people have to go offsite and register a new account to review scenarios, that's going to turn people off. Remember, we are trying to target people who are not invested. Those people are no likely to check out somewhere else. That's just the way it is. Let the Blades Forge do what it is good at: being a resource for designers.

Remember, simpler is better from the point of getting players to actually write reviews. Anything that these forums could not handle is too complicated in my mind. Nothing is as simple as write a 100 word or more review and rate it from 1-10. Requiring anything else will just scare people off.

People state a goal of expanding the base of players and, by consequence, the designing community. This requires a critical mass of participants and little else. Success breeds more success. Complicated systems located in a place where "uninterested" people are unlikely to go is not going to help this.
Link to comment
Share on other sites

Well, I guess we have difference philosophies, *i. To me, actions have consequences, and an abuse of the reviewing permissions would have the appropriate and negative consequence of a removal of privilege. And you seem to want to just protect us from ourselves. No offense intended, but that isn't really the message I think a person ought to be sending here.

Link to comment
Share on other sites

I agree with Stareye that it's easier to grant additional privileges than to take them away. I do, however, like Salmon's idea. I say we go with Stareye's idea first and then loosen it up if the need arises.

 

Anyway, I'm beginning to agree that having the rating system on this forum might be better, but I still like the idea of splitting the score.

 

Perhaps we can split quality scores as I described (Overall 1.0-10.0; Creative Content, Gameplay, Aesthetics & Functionality 1-10 + N/A and Abstain/No Opinion), but replace my two-way Genre ratings with the genre tagging idea others have suggested, where the scenario author tags it, or if that's implausible, experienced community members tag it, and then if the reviews are pushing heavily in a different direction it can be modified.

 

I think this would be simple enough not to scare anyone off, but specific enough to get more useful, more calculable ratings than a single flat score can give. It would require a bit more work on the part of the moderation to calculate these things, but I perhaps a script of some sort could work around that and make it easier.

 

In the opening post of each scenario thread, in addition to the author and title of the scenario, we could include this:

Quote:
Here are the required ratings, with higher scores meaning better quality. Enter a response for each field.

 

Overall Quality (1.0 - 10.0 to one decimal place):

 

(All the following have these rating choices: 1 - 10 with no decimal places, Not Applicable, or Abstain.)

 

Creative Content:

The artistic merit of the scenario. This category encompasses storyline, plot, themes, dialogue, descriptions, cliché or lack thereof, etc. This does NOT include spelling and grammar.

 

Gameplay:

What the player actually does while playing the scenario. This category encompasses combat, puzzles, balance (how well the combat, treasure, EXP, etc. interact), novel gameplay elements, ease of determining where to go and what to do, etc.

 

Aesthetics & Functionality:

The presentation of the scenario, things whose primary effect is to enhance (or distract from) immersion in the scenario. This category includes graphics, spelling and grammar, town design, cutscenes, effective programming, bugginess, etc.

 

After the above ratings, you must include a textual review of the scenario, in no less than 100 words.

That way the original post can simply be quoted, and give a standard and easy-to-use template in which to post numeric ratings, with the textual review below. It might make implementing a script to lift the scores and aggregate them easier too, so we don't have to rely on BainIhrno being available whenever we need him forever.

 

Any thoughts?

Link to comment
Share on other sites

Originally Posted By: The Almighty Doer of Stuff

In the opening post of each scenario thread, in addition to the author and title of the scenario, we could include this:
Including it verbatim in every thread seems silly; I'd suggest stickying it, but...

Originally Posted By: The Almighty Doer of Stuff

That way the original post can simply be quoted, and give a standard and easy-to-use template in which to post numeric ratings, with the textual review below.
...that would sacrifice this convenience.

It seems like a trade-off; which is better? Keeping it in one place, or making it as convenient as possible? (Keep in mind that if it's in a stickied thread you could stick quote-copy-paste it.)
Link to comment
Share on other sites

ADoS, and stareye - what are the criteria under which the privileges will be loosened? Is there something we can see? Some level of compliance after which confidence will return? I would like to see that, but I think it will be a difficult standard to employ. On the other hand, it is simple to set a standard for removal of privilege. Simplicity is good.

Link to comment
Share on other sites

The privileges will be loosened if we deem it necessary. It's impossible to foresee the future, so I don't want to set standards that may have to be revised later. It's not about confidence in people, it's about structure, utility, and cleanliness. Again, a lot easier to not let a mess appear than to clean it up and have people complain.

 

Of course, people still get to post whatever they want with respect to opinions in the relevant Blades forum (so long as it is within the CoC). If someone wants to discuss a review, have at it. If someone wants to give an opinion on a scenario, fine. The Review forum is simply meant to be a clean publication that anyone can contribute to.

 

ADoS: We tried doing things like "creativity" and "innovation" in one of the scenario contests. Long story short, it caused more harm than it did good. Why? Because these things are very subjective to begin with. No one could agree what these things even meant and not every good scenario fit into the "box". In fact, the numbers individually were pretty worthless. That's why future contests have been about one comprehensive number. It's still subjective but rating overall rating, but it seems more people can agree on "good" and "bad" in the overall sense than when you ask them.

Link to comment
Share on other sites

Then I am in favor of leaving the CSR in the democratic state of Shadowvale, as there seems to be no functional need for it to be removed, and to do so will restrict free posting. The CSR is supposed to be about everyone having the ability to post their judgment on a scenario, without having to wonder if it passes a test for which there are no guidelines. Sorry.

Link to comment
Share on other sites

Quote:
I'm in favor of Stareye's proposal, but some guidelines do need to be set.


Of course. The guidelines proposed are as follows:

1) The review must be at least 100 words.
2) The review must make a reasonable attempt to explain how the reviewer felt about the scenario and stick to the scenario being reviewed. In other words, be topical.
3) The review must meet be legible/understandable. No chatspeak or poor grammar.
4) The review must not attack the author (or anyone else) personally and adhere to the Code of Conduct.
5) The review must supply a score from 1-10 at the end.

The moderator may deny a review only if at least one of these guidelines is met. The moderator must then inform the reviewer of his or her decision, the reasons for denial, and offer constructive suggestions for how to address the moderator's concerns.
Link to comment
Share on other sites

Originally Posted By: *i
ADoS: We tried doing things like "creativity" and "innovation" in one of the scenario contests. Long story short, it caused more harm than it did good. Why? Because these things are very subjective to begin with. No one could agree what these things even meant and not every good scenario fit into the "box". In fact, the numbers individually were pretty worthless. That's why future contests have been about one comprehensive number. It's still subjective but rating overall rating, but it seems more people can agree on "good" and "bad" in the overall sense than when you ask them.
I quite like his outline, though. Perhaps it could simply be stickied as a suggested outline, with a note that it may not apply to some scenarios?
Link to comment
Share on other sites

*bump*

 

So are we all in somewhat agreement? Are we going ahead with *i's plan to bring the CSR here? Or adopt ADoS's method?

 

I am voting for the former. After reading this thread, I am no longer sure that there needs to be much change to a way a scenario is reviewed. As long as reviewers are thoughtful and helpful in their reviews, I guess then everything will fall into place. AND, having the CSR here is a great way to bring about more activity in BoA here.

Link to comment
Share on other sites

It's hard to fuel a discussion when the only people interested realize that no one cares what they have to say. I hate to sound cynical, but when you've got a bunch of people with their own unique idea of what the problem is, it's going to be hard to make any sort of progress. The original discussion really should have been something like "hey, what's the problem with the CSR." When you talk about revamping the CSR, it makes it sound like it's commonly accepted that the system is flawed, which some people, like me, don't agree with.

 

You'd need a list of issues to consider, and then come up with an organized plan. What we have, instead, is a giant mess of ideas that address undefined problems. That's how I see it anyway. I'm admittedly apathetic to this situation and wish I was a little more eloquent.

Link to comment
Share on other sites

New voice, entering here:

 

What exactly is going to happen when you do decide that you're "done" and that the discussion has ended in some sort of consensus? Everyone is free to create a CSR wherever someone will let them.

 

I can't speak for Stareye, but I'd imagine the question he wants answered right now is, "If I create a CSR on the Spiderweb Boards in the way I have been describing, will you all help write reviews and/or help moderate it?"

 

Just because someone creates a forum for something doesn't mean it will succeed; the point of this discussion is to convince people that a given proposal will work, and to convince them enough for them to agree to write reviews.

 

Originally Posted By: Naughty Salmon
So, is this a situation where the community gets to make a decision?
Yes, insofar as you agree to write reviews or not.

 

To everyone: if you like Stareye's proposal, now would be the time to say "I will contribute to the CSR as you have outlined it; I will write reviews." That goes for anyone's plan actually. If you like it enough to contribute, say so, because that is what determines whether it will survive.

Link to comment
Share on other sites

Originally Posted By: wz. As
I can't speak for Stareye, but I'd imagine the question he wants answered right now is, "If I create a CSR on the Spiderweb Boards in the way I have been describing, will you all help write reviews and/or help moderate it?"

The answer, for me anyway, is yes. I'd even moderate the BoA one and would help transfer SV reviews over to it.
Link to comment
Share on other sites

I'll admit, I'm against stareye's proposal for bringing the CSR to Spiderweb for one reason: this system of jumping through hoops that reviewers will have to endure in order to review a scenario. As fas as I can remember there have never been problems with flaming, or personal attacks, or spamming, in a review thread, and so these precautions are unnecessary. I do see that having to post a review through a moderator is a good way to ensure that the review is 100 words long and topical, and whatever else people wanted, but I also think that a lot of people will be turned off by this idea - whether or not it's a number comparable to the number of people who are put off by off-site reviews, I don't know.

 

If the community (and Jeff: Ephesos raised a really good point here) wants the CSR here, then I'd like to be trusted by the administration/moderation team to be allowed to review without having to submit it for approval. If they feel we can't be trusted to be responsible reviewers, I vote that we keep the CSR at Shadowvale.

Link to comment
Share on other sites

Originally Posted By: Kelandon
If we continue trying to come up with a single quality number for scenarios, I will continue to stay away from CSR and request that my scenarios (or anything that I have anything to do with) not be on it.
While a single quality number can hardly be considered accurate, it's still useful as a vague measure of how good a scenario is.

Originally Posted By: Nikki.
I'll admit, I'm against stareye's proposal for bringing the CSR to Spiderweb for one reason: this system of jumping through hoops that reviewers will have to endure in order to review a scenario. As fas as I can remember there have never been problems with flaming, or personal attacks, or spamming, in a review thread, and so these precautions are unnecessary. I do see that having to post a review through a moderator is a good way to ensure that the review is 100 words long and topical, and whatever else people wanted, but I also think that a lot of people will be turned off by this idea - whether or not it's a number comparable to the number of people who are put off by off-site reviews, I don't know.
I don't see how you'd be jumping through hoops. As I understand it, you would simply post your review in the relevant thread as normal, but it would not show up until some moderator approves it. This effectively makes it impossible to post anything other than reviews in the thread. And as I understood it, all reviews would be approved unless they were one-word or one-sentence reviews, or two-sentence, or I guess anything less than a decent paragraph.
Link to comment
Share on other sites

I would use Stareye's system, but I would prefer it if Salmon's idea were implemented immediately.

 

You could just say something in the rules like, "If you post X small number of consecutive, rule-adhering reviews (or have already posted such on the Lyceum/Shadow Vale CSR) without needing corrections, you are granted the privilege to post reviews freely. Be warned, however, that this privilege can be taken away if your subsequent reviews show a tendency not to adhere to the rules, and will only be regranted at moderatorial discretion."

 

Minor corrections can be sent via PM without taking away the privilege, of course.

Link to comment
Share on other sites

Originally Posted By: Celtic Minstrel
While a single quality number can hardly be considered accurate, it's still useful as a vague measure of how good a scenario is.

Maybe you're saying the same thing as I am here, but I disagree with the outcome. Yes, knowing that something is rated a 2 gives me the appropriate expectations going in, but I've regularly liked things rated in the 6-7 range and hated things rated in the 9+ range. There seems to be a boundary somewhere near a 3 or a 4, below which the scenarios are different from the scenarios above it, but I can't say that I've ever otherwise gleaned anything useful from the numbers.

Okay, let me cite examples. The difference between an 8 and a 9 is essentially negligible, as far as I'm concerned. I liked Emulations and NTH, but I liked The Election much, much more, and it's rated a full point lower. Being rated as low as a 5 is sometimes meaningless; I liked Hunted, SPS, and Erika's Legacy maybe just a little less than I liked Emulations and NTH (though I'll grant that I never finished Erika's Legacy, but unlike any Alcritas scenarios, I didn't just get annoyed the first time I came across any combat and give up). I'll grant you that Ravaged Land (a 1) was pretty freakin' bad, but other than that — and this is my fundamental problem with CSR — the numbers on CSR don't predict anything about my reaction to a scenario.

My point here is that the Spidweb method ("These scenarios are solid, these scenarios are generally not, and these scenarios we don't know about") is a lot more valid in theory (if not in practice) as far as predicting my reactions to scenarios. Having a 1-10 system with decimals is ludicrous. Maybe a 3-star system or something, but anything more nuanced is bound to obscure more than it reveals.
Link to comment
Share on other sites

Originally Posted By: The Almighty Doer of Stuff
I would use Stareye's system, but I would prefer it if Salmon's idea were implemented immediately.
I forget what Salmon's idea was...

Originally Posted By: The Almighty Doer of Stuff
You could just say something in the rules like, "If you post X small number of consecutive, rule-adhering reviews (or have already posted such on the Lyceum/Shadow Vale CSR) without needing corrections, you are granted the privilege to post reviews freely. Be warned, however, that this privilege can be taken away if your subsequent reviews show a tendency not to adhere to the rules, and will only be regranted at moderatorial discretion."
I'm pretty sure Stareye said anyone can post reviews freely; they just won't show up instantly.

Originally Posted By: The Almighty Doer of Stuff
Minor corrections can be sent via PM without taking away the privilege, of course.
This doesn't make sense; what does a private message have to do with it?

Originally Posted By: Kelandon
Originally Posted By: Celtic Minstrel
While a single quality number can hardly be considered accurate, it's still useful as a vague measure of how good a scenario is.
My point here is that the Spidweb method ("These scenarios are solid, these scenarios are generally not, and these scenarios we don't know about") is a lot more valid in theory (if not in practice) as far as predicting my reactions to scenarios. Having a 1-10 system with decimals is ludicrous. Maybe a 3-star system or something, but anything more nuanced is bound to obscure more than it reveals.
Well, okay, a simple "bad/decent/excellent" system would work... or three to five stars or whatever...

P.S. If we made a star-based rating system, the stars should probably be added as emoticons. laugh
Link to comment
Share on other sites

Salmon's idea was that certain trusted members would be given free posting access to CSR without having to go through a moderator.

 

Actually, now that I think about it, I'm not entirely sure what benefit it gives anyway. The only reason I can think of that would make post-by-post moderation a significant inconvenience for the poster would be if they intended to break the rules anyway. I don't think the rules are at all unreasonable, so I don't see what the problem is.

 

So I'm fine with Stareye's idea, with or without Salmon's idea.

Link to comment
Share on other sites

Originally Posted By: The Almighty Doer of Stuff
The only reason I can think of that would make post-by-post moderation a significant inconvenience for the poster would be if they intended to break the rules anyway. I don't think the rules are at all unreasonable, so I don't see what the problem is.


I really dislike this argument. Really.
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...