petec, that's
really not what standard deviation does.
Standard deviation is a measure of how much, on average, each score deviates from the statistical mean (= the total scores / the total number of scores, not a hypothetical average taken by sticking a pin in the centre of the range of possible numbers).
In your example, both Writer A and Writer B would have
exactly the same standard deviation. Let me demonstrate:
Writer A scores 100 scores of 1 & 1 score of 3
In total he has (100 x 1) + (1 x 3) = 103. His mean is total score / number of scores = 103 / 101 = 1.02 (to 2 decimal places).
The 100 scores of 1 each deviate from the mean by (value - mean) = 1 - 1.02 = -0.02. Remove the minus sign here because otherwise our calculations will explode. So it's 0.02. The 1 score of 3 deviates by (value - mean) = 3 - 1.02 = 1.98.
The variance is the mean of the squares of these deviations, so (100 x [0.02^2]) + (1 x [1.98^2]) = (100 x 0.0004) + (1 x 3.9204) = 0.04 + 3.9204 = 3.9604, divided by the total scores 101 to get the mean, so 3.9604 / 101 = 0.0392.
The standard deviation is the square root of the variance, so root 0.0392 =
0.2 (back to 2 decimal places).
Writer B scores 100 scores of 5 & 1 score of 3
In total he has (100 x 5) + (1 x 3) = 503. His average score is total score / number of scores = 503 / 101 = 4.98 (to 2 decimal places).
The 100 scores of 1 each deviate from the mean by (value - mean) = 5 - 4.98 = 0.02. The 1 score of 3 deviates by (value - mean) = 3 - 4.98 = -1.98 and again we remove the minus sign, so it's 1.98.
The variance is the mean of the squares of these deviations, so (100 x [0.02^2]) + (1 x [1.98^2]) = (100 x 0.0004) + (1 x 3.9204) = 0.04 + 3.9204 = 3.9604. Divide by the total number of scores, 101, and you have 3.9604 / 101 = 0.0392.
The standard deviation is the square root of the variance, so root 0.0392 =
0.2 (back to 2 decimal places).
If I might borrow Old Hack's post:
Various explanations were given about how the rankings worked: the scoring system relies on ignoring deviations from the norm (standard deviations).
The way this works is to state, for example, "Any score which is over 2 standard deviations from the mean is discarded".
For Writer A, the mean is 1.02 and two standard deviations is 0.2 x 2 = 0.4. So any score which is
above (1.02 + 0.4) = 1.42 or
below (1.02 - 0.4) = 0.62 is discarded. Because that value is a huge way away from the norm. Acceptable range is 0.4 to 1.42*.
For Writer B, the mean is 4.98 and two standard deviations is 0.2 x 2 = 0.4. So any score which is
above (4.98 + 0.4) = 5.34 (but the range tops out at 5, so we'll call it 5) or
below (4.98 - 0.4) = 4.58 is discarded. Because
that value is a huge way away from the norm. Acceptable range is 4.58 to 5*.
Standard deviation is an excellent way of determining which outliers are statistically abnormal and then discarding them. If you decide how many standard deviations away from the norm you'll accept in the beginning and apply the same calculation equally to everyone, there is
no bias in your results. Standard deviation doesn't even hurt polarising, contraversial work more than tame middle-of-the-road stuff, because polarised scores give a higher standard deviation and thus a wider acceptable range. The only possible shenanigans come where you set standard deviation parameters on some stories' ratings and not others, or apply different parameters to different stories, or whatever.
In short, using standard deviation is not in itself suspicious - personally I think it's entirely sensible in this context.
* (The range of acceptable values changes based on how many standard deviations you accept. In these cases the range seems very narrow because the starting data are deliberately skewed to present a contrast. A normal set of data would have provided a much wider range.)
I hope I didn't (1) kill the thread, (2) bore anyone or (3) betray my shaky grasp of GCSE-level mathematics here. Pesky standard deviation always threw me in my exams.