Just out of interest: On what kind of standards or criteria do you assess the specific qualities of the settings you review?
I don't mean the pretty obvious forms of the ranking but the ideas behind the different grades for the differnt categories? Are there some kind of objective gudelines or rules of thumb one could rely to when assessing a setting? Or is it just gut feeling in the end? Does it make sense to compare all settings beforehand and evaluate them then or should every contribution be seen on its own?
I used some basic questions for the evaluation, mostly "Do I like it (inspiration)?, How much feels familiar in a good way (there is nothing new under the sun after all, and thus the question is not how new something is but how well the probably already existing elemtns are assembled into something new and refreshing), but I have my problems with the last category, so it mostly breaks down to "how easy can the text be followed and how much fun does it make to read it?
Relative Quality
It's important to see the settings in comparison to each other. I give 5 points per category to the ones I think best in this regard, 1 point to those I think the worst, and the rest in between on a relative scale.
The other criterium I think is important is the support of the setting after the original presentation. This is about a work in progress and therefore the more progress a setting shows and the more the original author reacts on criticism and suggestions from the reviews, the better.
I don't know.
Shouldn't every setting be seen on its own and measured by its own specific qualities and flaws?
Why?
The idea is to find the best setting. By treating any setting relative to every other one, you make sure that they are comparable and that the best setting ends on top.
Long Term
That is an interesting point, but it works best in the short run, not the long run. A person could have a great setting, but get a low score when compared to some settings that are even better by this method (which can lead to discouragement for some, irritation for others if they don't see a reason for the score) which may not be reflective of the setting in question. While in the short term this works becasue like you said we are just trying to pick the best, it leads to a few problems in the long run. The extremes, the ones who get all 1s or all 5s don't necessarily deserve those scores in each area and doing so drastically shifts their average as well as preventing the setting designer from knowing what needs to be improved. Also, if the first game does catch on, we might always reopening setting submissions and using the old setting submissions as well, in which case the old rankings wouldn't apply anymore, but would serve to scew the results.
Since you mention it...
If our model proves to be successful, the setting concept competition will definitely happen again - probably no earlier than in Spring 2011. If we used the same selection process, I will be zeroing out the previous reviews.
Okay
Aww, well then scratch my comments out then, Fenris method works alright then.
Setting evaluation
There are no hard and fast guidelines for the different qualities, it's really up to personal preference. Here are some ideas though.
Inspiration is how interested you are in doing development work on a setting. If you aren't interested in working on it at all, then you would give it a very low inspiration score. If you could see yourself writing something for the setting every day, then you would probably give it the highest score for inspiration.
For originality, you are correct that creating something that is truly unique is impossible. What we're going for is something that has not been done at all as an RPG, something that hasn't been done well as an RPG, or something that takes other RPG elements and does something very new with them. Dragons forming motorcycle gangs to manipulate the drug trade would be very original, but it also probably wouldn't be very inspiring. ;)
Writing quality is exactly as you describe with spelling and grammar thrown into the mix. Are the thoughts coherent? Can you figure out what the author is trying to do? Do you think they did a good job describing their concept? Is it relatively free of grammar and spelling mistakes?
When I wrote my reviews, I tried to think in my mind how the different concepts compared to each other and not just how each one alone was done.
Sidenote
As a sidenote, I think the voting process is a bit intransparent. You mentioned that the different categories are weighed differently for determing the final score, but I couldn't find the actual weighting.
Transparency
I made it purposefully not transparent to avoid anyone trying to game the system. I will say that inspiring is the most significant followed by originality and then writing quality.
Originality and Writing Quality
For the originality, I think it is important how much different elements the setting include and combine. The more facets it has and the more different elements it includes the better. It should also be seen as a measure for how many original campaigns you can play with it, and how diversified it is.
For the writing quality, I would also suggest to take a look on the length. The longer, more detailed and informative a desctiption is, the better. I don't mean when people just can't come to the point and write on and on, but how much explanation they add to the setting and how well it is structured.