Contest Discussion

[*i]

Eligibility

 

Both Blades of Avernum and Blades of Exile scenarios are eligible for entry into the 8th Scenario Design Contest with the following guidelines:

 

1) The scenario cannot have been an entrant in any previous Spiderweb of Olympia scenario design contest.

2) All entered scenarios (utilities are exempt) must have been submitted to Spiderweb Software.*

3) The deadline for entry is November 30, 2006.

 

* If you already submitted to Spiderweb and either have not been posted or have been removed, you may host it independently. However, you must have actually attempted to submit it to Spiderweb software.

 

Categories

 

Small Scenario

Large Scenario

Newcomer

Best Utility

 

Small Scenario: < 25 zones

Large Scenario: >= 25 zones

 

Zones is the total number of towns plus outdoor sections. Near identical towns (such as use of VTE) count as 0.5 of a zone.

 

To be eligible for the Newcomer category, you must not have been a participant in any previous Spiderweb or Olympia scenario design contest. If this applies to you and you have more than one entry in this contest, you may select only one to be in the Newcomer category.

 

Judging

 

Initial Rounds

 

Scenarios in each category will be given a score from 1-10 based on quality, 1 being lowest and 10 being highest. The decision criteria is generally up to the individual judge so long as he/she is consistent where possible. The exception is that a judge cannot favor either BoE scenarios over BoA scenarios or vice versa, each must be judged on it's merits. All judges scores are averaged to make a cumulative score.

 

The highest ranked scenario in each category (except Best Utility) and any scenario with a score of 80% the highest score in the category will advance to the Best Scenario round. For example, suppose a category has four scenarios ranked 8.8, 8.5, 7.6, and 6.3. The cutoff for admission to the final round is 7.04 (0.8*8.8), so the top three scenarios advance and the lowest one does not.

 

Final Round

 

The scenarios that advanced are judged in a final best scenario round. All participating judges then discuss and debate to rank the scenarios from best to worst. Each judge is given three votes that he/she may distribute as he/she sees fit with a maximum of two for one scenario. How the judge actually allocates votes are secret.

 

Once voting has completed the lower 50% rounded down (if there are 7, only the lowest 3 are dropped) are eliminated. If there are ties within the cutoff, all tied scenarios remain. Any votes for eliminated scenarios may be redistributed at the judge's discretion; votes for scenarios still in play may not be reallocated. The process continues until one scenario has the most points.

 

Tie Breaking

 

In the event of a tie that cannot be broken because no votes remain to be reallocated, with the exception of the final decision round, the cumulative score of the initial round is used to make the decision. Should those scores be tied, the high scores are compared and on down to second highest, third highest, and so on. In the event of an actual tie between all scores, both (or all those tied) scenarios are eliminted.

 

If a tie occurs in the final decision round (i.e. two scenarios remain), all votes are cleared and the judges vote for their preferred scenario. If this vote ends in a tie, the above criteria are used. If all criteria are tied, the competition is a tie and both are co-winners.

 

In the event more than two scenarios remain and all remaining scenarios are tied, the original decision criteria is used until two remain (at which a vote is done as above) or the tie is broken. Should the tie be impossible to break, a vote will be done to attempt to break a tie. If doing so is impossible, then all scenarios that are tied are declared co-winners.

 

* * *

 

Comments?

[Swimmin' Salmon]

Zones, towns, sections. It seemed to folks in the discussion that what mattered for size was the depth of the story. Empty outdoors was small, while a town that changed 6 times over the course of the scenario was large. That is why we talked about adding together non-dialogue scripts to come up with the number of zones.

 

Also, is Best Utility still a category? I remember some discussion on how to best include things like the 3D Editor and Alint.

[Drakefyre]

I'd add a best utility category, especially as we would have plenty of entrants.

 

I think that zones is better than scripts - the player still has to walk through 8 empty outdoors sections, which makes the scenario larger in terms of playing (but not designing).

[BainIhrno]

Quote:

Originally written by *i:
2) The entered scenario must have been submitted to Spiderweb Software.*

I'm sorry, but I'm not sure what you mean by this.

[*i]

I would agree with you on scripts as a metric except that a long way back we decided to have both BoE/BoA scenarios be eligible for entry. Since there is no analog for scipts in BoE, it would not be very workable without splitting categories which would make for small categories.

 

Having two types of scenarios allowed will raise interesting issues as far as judging goes. We'll have to iron those out when we see a list of entries. We may have to split the judging based upon BoE/BoA accessibility. Further comments?

 

Btw, does anyone have a list of eligible scenarios?

[*i]

Quote:

I'm sorry, but I'm not sure what you mean by this.

Sorry for the double post. What it means is that if you have a scenario, you must submit it to the tables on Spiderweb Software or at least attempt to.

[BainIhrno]

Elgible Scenarios (all of these have been released)

 

Small:

A Perfect Forest/BoA

The Final Spire/BoE

Redwall: Prologue/BoE

A Good Beginning/BoA (?)

The Foolish Giant/BoE

Nephilim Mystery/BoA

Proving Grounds/BoA

 

Large:

Magus of Cattalon/BoA

Dilecia/BoA (?)

 

Utility

3D Editor

Alint

HLPM (Kelandon)/BoA

Fifth Tower of Magi/BoE

Portal/BoE

Ultimate Equipment/BoE

[*i]

APF would indeed be eligible and will be submitted for those wondering.

[Lazarus.]

Dilecia has 16 zones, so its a small, although since it'll be in Newcomer I don't suppose it matters.

 

On that note, is a scenario entered only in the Newcomer category? Or is it entered in both Newcomer and the appropriate category for its size?

[Drakefyre]

Both categories.

 

Don't forget the graphics shifter for utilities.

[*i]

Nominally I would say both as well. Of course, we could limit it to one if we want to make things easier on judges. Speaking of judges, accessibility issues creates an interesting challenge. I propose the following:

 

A member may judge a category so long as he/she does not have a scenario in that category. He/she may judge only BoE scenarios, only BoA scenarios, or both depending on the judge's access to software. If the judge does both, he/she must agree to not favor one media over the other -- scenarios must be judged on their individual merits regardless of format.

 

An interesting idea came up that we would also have a BoE and BoA category in addition to the previous ones. This would try to eliminate bias towards one. So basically you would be entered into all eligible categories. Comments?

 

Also, looking at what we have, we may need to reformulate the category definitions. Does anyone know if the bar were lowered to 20 zones if it were to make a difference?

[Drakefyre]

Well, there are three large scenarios in beta-testing: Exodus, E:R, and where the rivers meet.

[*i]

Fair enough, I think a good approximate ratio to get to is about 2/3 small, 1/3 large.

[BainIhrno]

Quote:

Originally written by *i:
An interesting idea came up that we would also have a BoE and BoA category in addition to the previous ones.

Yeah, I like that idea too (well, of course. ). I was thinking about stating this myself, in fact.

[Drakefyre]

TM and I were talking about three categories: BoE, BoA small, and BoA large. That is, if there are problems getting judges as-is.

[Niemand]

For utilities: GraphicAdjuster (which I think Drakefyre meant by 'the graphics shifter') and DialogEditor. However, these and other utilites may not fit well with the Submitted to Spiderweb rule.

[*i]

Utilities are now exempt from submission as they are another type of beast. However, I think it's only far that Spiderweb get to host the scenarios -- that and it makes judging a bit easier with a central download place.

 

As far as the BoE/BoA categories, I think I'll defer the decision until we have a better idea of the submission and judging situation.

[*i]

*bump*

 

Any more suggestions on the rules? As I said previously, I think the categories are good for now until we see exactly what we have. Until then, any other comments on rules or anything left out?

[Drakefyre]

I'm not so keen on the point judging system. I wonder what a relative ranking system would give ... someone (maybe me, later) could go through the last contests on the Lyceum and see how the contests would turn out if they were judged in the following ways:

 

MLB MVP voting

Instant-runoff system

Condorcet method

 

EDIT: A reason for this is to prevent a few people who hate or love a scenario from dragging a scenario to victory or defeat.

[Ash Lael]

As I have mentioned earlier, I much prefer the concept of discussion judging to numeric scores.

[*i]

I don't like Drakey's idea at all. The primary reason is that I feel the past method of doing things based on a point scale appear to work quite well and be the most fair. For this reason, I say we stay with the point system and I'll give an example of why the methods proposed really don't work well here.

 

Suppose there are three scenarios in an initial bracket (say Small Scenario), two are of "about equal" superior quality and one not so great. A judge is forced to rank a top, middle, and bottom. Essentially the scores translated to a 0-10 scale become quantized to 10, 5, and 0. A judge MUST assign one of each, he/she cannot give any measure of relative quality at all other than a pure ranking. Under this system, you can unfairly penalize scenarios of equal quality.

 

At the end of the day, one scenario would advance to the final round and two would not (I'd assume top 67% would be a bit much). Under the current system, the top two would advance if they were of roughly equal quality.

 

Basically, we shouldn't penalize a really good scenario for just so happening to be in a bracket with one of similar quality. Essentially, you will have really good scenarios that will not be able to advance to the final round based essentially on a coin toss of who is judging.

 

Once we get to the final round, such things might be workable as there needs to be a clear winner. But even here, we're really limiting the ability of the judges. Suppose I judge and I really like scenario A and B, but not C. I have to unfairly penalize A or B at the expense of the other rather than just assigning them relatively equal scores. Yeah, either A or B will win in the end, but at least I'm free to rank them near equally.

 

As far as the few people dragging things down argument, I don't see how it helps overly much as judges could conspire and still be unreasonable in their rankings. Also, to the notion of a few people hating a scenario, I say too freakin' bad. We design scenarios for the community. The judges aren't just pie in the sky scenario experts, the judges are part of the intended audience -- the community. If there is a section of people who don't like it, they have as much right to their opinion as anyone else and may give their scores accordingly.

 

Discussion judging -- Interesting and could work. I'm a little leary of it as I feel it's better to have several independent measurements rather than judges discussing and persauding. Maybe this is my bias as a scientist, but I just feel uneasy about it.

 

Quote:

someone (maybe me, later) could go through the last contests on the Lyceum and see how the contests would turn out if they were judged in the following ways:

One more thing, I'd say differently. However, that does not necessarily make the answer more right, just different. I could use the argument had we selected different people to judge as well. Besides, had we used a different system, perhaps people would have judged differently. Basically, any results these tests would give would ultimately be irrelevant as it comes down to opinion anyway.

 

EDIT: Ash's point made me think of something else. I should add that I think judging needs to be done in relative secret until all results are in, at which point the scores and comments would be posted. This keeps the judges as independent as possible.

[Drakefyre]

I've just done my tests on the 7th contest - it looks like we don't have enough judges for Concordet or Borda - one vote can change things a lot. The results were the same except for Roots and NTH switching (by one point) and Corporeus and Areni switching (also by one point). IRV gives the same results as numerical, but I can see where it has advantages over the numerical system.

 

EDIT: I should add that this was for Best Scenario judging - which happened to be every scenario entered.

 

And I'm not sure I like the discussion judging for something like this - independent judges tend to give more real, unpressured results.

[*i]

Quote:

but I can see where it has advantages over the numerical system.

For those of us lacking your acuity for these things, could you enlighten?

Also, your results are meaningless regardless of what they are for the reason I stated above. They are only different, not better.

[Drakefyre]

Well, IRV results aren't different at all. The advantage I see is that it's a straight majority vote for best scenario - in a points system everyone can rate a scenario an 8.5 and none of them have it 1st choice, and it will still win. Nobody (except the author) is happy then. After second/third votes are counted for getting a majority for best scenario, it ensures that the best scenario is actually chosen.

[*i]

I guess we have a different idea of "best scenario" then. I consider "best scenario" to be a cumulative overall thing, not an individual thing for the purposes of these contests. It doesn't have to be anyone's favorite to be "best" it just has to be the most liked overall.

 

I know given the situation of relatively equal choices, I would have a hard time ranking things with as better or worse. I, as a judge, would rather say they're equal and move on.

[Drakefyre]

I guess another part of my disillusionment with the ranking system is that the contest results are basically predetermined based on CSR ratings. This would be a vote for a discussion-based system here, actually. Or at the very least having all scores emailed to the contest runner, so nobody sees other judges' scores until after the deadline to maintain independence.

 

EDIT: Yes I know these are completely different ideas, but they're somehow similar in my mind.

[Swimmin' Salmon]

If I do get the honor of judging, I would like there to be be several criteria for scoring a scenario. Has this been discussed already? I'm talking about atmosphere, presentation, scripting "wow"s, etc. Typically when grading a multifaceted thing like a scenario, you can assign letter grades across each category, and then let the judge assign a weighting to that scenario. For instance, if scenario X is really well scripted (A) and is flowing logically (, but the atmosphere and graphics (E) just make you want to stab your eyeballs, the judge could then heavily weight the atmosphere, award an overall C, and explain in their comments how that judgement was reached. The next judge may not care so much about the graphics, and prefers the story is really cool, weighting that more and having it end up as a B.

 

An arbitrator could compile the weighted scores for each scenario (C and B above) and average them, awarding a numerical winner that is based on player/judging preferences.

 

I prefer to think of this as the Complexosimpleton Methodology of Thought Transferance and Complication.

 

Or CMTTC for simplicities sake.

[*i]

The problem with rubrics, as debated many a time by Alcritas and Brett Bixler et al. in the days of antiquity, is that not all scenarios fit the "mold".

 

Building a rubric makes certain assumptions about what a scenario should be like. Unfortunately, it's impossible to make a rubric that is sufficiently detailed enough to be useful while accounting for every creative good idea out there.

 

An example is the issue of outdoors. Some rubrics specifically asked the judge to rate outdoors. This is fine and dandy until you get to a scenario that has no outdoors. Do you penalize an otherwise superior scenario or do you modify the rubric accordingly?

 

Hopefully you would do the latter. Granted, that was an easy one. Unfortunately, in doing so for every possible subtle case you reduce to a final point of just rate the scenario. Of course, you as a judge are free to make whatever criteria you want so long as you are relatively consistent where possible, use 1-10 as the quality scale as described, and do not favor either BoE or BoA scenarios over another.

 

On another topic -- For the record, the ballots should be secret until judging is complete. This is the one way I would conduct the contest differently.

[Swimmin' Salmon]

The identity and temporary mailing addresses of the judges should be available prior to judging if there is to be any reliable method of bribery.

 

Just sayin', ya know?

[Drakefyre]

Salmon's method reminds me a little of staff picks at a book or movie store.

[BainIhrno]

I don't like the rubric idea at all, and I mostly agree with *I on this one.

 

I like the idea of discussion-based judging for Best Scenario. Not for preliminary judging, though, since doing that would probably be more trouble than it's worth.

[Swimmin' Salmon]

I rate stareye's judging methodology with 5 snowy plovers. (Extra point for fawning support of Bain )

Drakefyre's method is awarded 2.5 rusty nails in an old tin can.

 

By popular choice, as well as sheer momentum, stareye wins the 8th People's Choice Awards. Drakefyre now must straighten out his nails and pound them into the snowy plovers and cook dinner.

 

(Please, someone get me coffee!!!)

[Jewels]

Consider Redwall: Prologue entered for Newcomer, Small, and BoE if you do it that way.

 

I've sent it to Spiderweb, but they haven't sent me a confirmation e-mail yet. It's available at TrueSite if they don't get to posting it.

[Lazarus.]

I like the idea of judging on a 1-10 scale in the preliminaries, then discussion to decide the final round. The only real problem I have with scoring on a 1-10 scale is that scenarios often seem to receive the same score as they had at the CSR. Which could suggests that judges just take the score they see at the CSR and modify it by half a point or so depending on their personal feelings. Or it could be because the judges are the only people rating it at the CSR, so the scores are just duplicated

 

Quote:

Originally written by Jewels:

I've sent it to Spiderweb, but they haven't sent me a confirmation e-mail yet.

Spiderweb doesn't give confirmation e-mails, at least I never got one.

[Drakefyre]

Right - before CSR existed numerical scores were fine, but now they're just a mirror of CSR. Discussion judging for the finals would be good for me.

[*i]

Integrated some of the suggestions. The tie breaking rules are a bit complex, but should be all inclusive and will hopefully not need to be used.

 

Please comment...

[Drakefyre]

When it says you have three votes to allocate, can I place all three on the same scenario?

[*i]

You may, should you strongly favor one over all the others. If this is too abusive, we could max it at two per scenario.

[*i]

Sorry again for the double post, but this is a new issue altogether:

 

With the change in rules, it makes it a requirement for those in the final round to have access to both BoA and BoE. I foresee this creating problems. A possible solution is to break the contest into two, one for just BoE and another for BoA.

 

Just out of curiosity, how many people out there have access to both?

[Kelandon]

Quote:

Originally written by *i:
A possible solution is to break the contest into two, one for just BoE and another for BoA.

I would be totally in favor of this, if the number of scenarios justified it. I don't think that it will, though.

[Ash Lael]

Quote:

Originally written by *i:
Sorry again for the double post, but this is a new issue altogether:

With the change in rules, it makes it a requirement for those in the final round to have access to both BoA and BoE. I foresee this creating problems. A possible solution is to break the contest into two, one for just BoE and another for BoA.

Just out of curiosity, how many people out there have access to both?

I do, and will be judging.

[the Great Potato]

Right now I only have access to BoA, and if I get access to BoE (which I'm trying to), then there's a possibility I could only judge newcomer and large.

 

EDIT: Ash, does that mean you'll have no entries in the contest?

[Jewels]

Quote:

Originially written by Spiderweb Software:
We always send a confirmation E-mail after getting a scenario. If you don't get one, something has gone wrong ... E-mail us.

At least it says they do on the Submission Page. It's not on the Untried table at any rate.

Quote:

Originally written in the rules:
In the event of an actual tie between all scores, both (or all those tied) scenarios are eliminted.

Is this new? You can't possibly do this. Staying with the same theme, there are three scenarios in a category, two are basically equal and one is terrible. If the two that are equal result in a real tie, they are both eliminated and the terrible scenario wins.

Suggestions for change:
Appoint an official tie-breaker.
Use CSR's averaged scores.
Use CSR's highest score given.
Just give them both first(or whatever they tied for) and skip giving a second(next) place.

I like the 0-10 ranking for regular categories, but the best to worst for final round.

[*i]

You took the sentence out of context. This is after all other metrics have failed. This means that the number of vote points is tied, both cumulative scores of the initial brackets have tied, and all other scores given by the judges are tied. This is a highly unlikely chain of events and will probably never happen. This is what is meant by an "actual tie".

 

Also, it only effects things on the "bottom" where things are to be eliminated. If the top and second place scenarios are tied and there are nine scenarios, only the bottom four are affected. This is where, in this case, there is difficulty in determining who is in fifth or sixth.

 

Finally, the tie breaking is only applicable in the last scenario round. In the inititial rounds, we can have all scenarios advance if necessary so this issue does not come up as there are no eliminations. In the event of a tie in the initial rounds for the top spot, we'll use the highest score criteria and on down. If all match, it is a true tie. Otherwise, it's irrelevant.

 

Again, the tie breaking rules are there to account for unlikely, but potentially annoying situations that could hypothetically come up. I tried to cover every possible case, so they can be convoluted.

[BainIhrno]

Quote:

Originally written by the Great Potato:
Ash, does that mean you'll have no entries in the contest?

I'm not Ash, but I can say he won't. We're working on a BoE scenario together, and given the progress rate, I can't see it being released until at least February 2007. Maybe even later.

Also, I don't like the idea of breaking it into two contests at all. Even if there are enough entries, that feels really sloppy.

Third, I don't fully understand the judging for best scenario. First you say discussion and ranking. Then you say by voting.

[Drakefyre]

I'm not too keen on having three votes to throw around - maybe you could make each judge pick a 1st and 2nd choice worth 2 and 1 points respectively? Or even 3 and 1?

[*i]

I don't feel we should force judges to make a first and second choice if they are equal. I don't see any real problems with the three votes other than we could cap it at two for one scenario. It would have the same effect essentially, but a judge is free to pick three choices.

 

Personally, I don't see too much of a problem either way. My purpose is to give the judges a reasonable amount of latitude.

 

Quote:

Third, I don't fully understand the judging for best scenario. First you say discussion and ranking. Then you say by voting.

What's there to not understand, people discuss/debate, and then they cast a private ballot?

[BainIhrno]

But what's the point of having the discussion if the judges just cast a private ballot? I suppose that's what I don't understand.

[Drakefyre]

So people can voice their opinions but people are not pressured to vote according to the more vocal judges. It helps people consider aspects of the scenarios they hadn't before.

[*i]

The judges have a chance to debate the others to influence the decisions of others and convince them of favorites and other issues surrounding the scenarios. Judging comments will be made public to judges at that point and they can be challenged/discussed by other judges.

 

The reason for a secret final ballot is to minimize strategizing and being influenced by actual votes. People may vote for things in an attempt to mess with the system. If the ballots are kept secret, none of the other judges know how the others are acting.

 

Just to add, I agree with Drakey here.