Testing the Wisdom of Crowds

On January 11th, I spoke at a gathering of the Libertarian Alliance about James Surowiecki’s The Wisdom of Crowds book (previous post here).

I got 15 responses to a questionnaire I handed around at the meeting (recording here) and can now report on my findings.

The questions were as follows:

Question 1. Regardless of what you think will happen, which political party or parties, if any, do you want to win the next British general election?

Question 2. Below is a list of six Premier League football clubs, please list them in order of closest to this place [near Senate House, Malet Street London WC1], to furthest (1 is closest, 6 is furthest. [Ranked in alphabetical order: Arsenal, Chelsea, Fulham, Hull City, Manchester United and Wigan Athletic]

Question 3. Guess my weight (in kilogrammes, stones or pounds). [I stood in front of the audience]

Question 4. Which of these four themes is part of the 2010 Spring Collection, according to the latest trends being pushed by fashion houses?
[Stiletto heels; Kitten heels; Pompadour heels; Achilles heels]

Question 5. Regardless of what you want to happen, which political party will have a majority after the next British general election? If you think no party will have a majority, write “None”.

I also asked respondents to rate how they thought the group would do as a whole. However, I have not reported these as it is clear from the responses that this instruction was not well explained by me (some responded in relative terms, others in absolute terms).

Responses and my comments below the fold…

Question 1.

Political Party Respondents percent (%) Seats in House of Commons*
Conservative 4 27% 236
UK Independence 4 27% 236
No One 4 27% (not voting)
Libertarian Party UK 2 13% 118
Liberal Democrats 1 7% 59
Labour 0 0% 0

* 1 seat undetermined.

Just to compare, current standings: Labour 346, Conservative 193, Liberal Democrats 63, various Nationalist and Unionist parties 28, UKIP 1 and LPUK 0, 12 others.

NB: Clearly an opinion pollster standing outside the meeting room would have had a shock if my audience had been asked “who would you vote for if the general election were held today?”

Question 2.
The correct order was the same as alphabetical (though I didn’t know this when I chose them).

Club Average estimate Est Ranking Correct rank
1. Arsenal 1.4 1 1
2. Chelsea 1.93 2 2
3. Fulham 2.87 3 3
4. Hull City 4.87 5 4
5. Man Utd 4.4 4 5
6. Wigan Ath 5.53 6 6

NB. I was very impressed with this result. Most people present were not football fanatics and most do not have a detailed knowledge of UK geography. I would have got this wrong. Only six people out of 15 did better than the crowd.

Question 3. I was personally pleased with this result, but quite surprised: no one overestimated my weight, which meant that the average was poor. I suspect that the response was like asking “Do I look fat in this?” This would not apply in the competition to guess the weight of an ox (cited in the book).

One person, using a faint marking pen, seemed to guess “8 stone” or 112lbs, which if true, would make me seriously underweight. I’ve therefore produced a mean result with and without the outlier.

Range 1: 165lbs to 252lbs (14 estimates).
Range 2 (with outlier): 112lbs to 252lbs (15 estimates).
Average 1: 209.5lbs
Average 2: 203lbs

Actual weight: 260lbs.

Error 1: 19.5%
Error 2: 21.8%

NB. Two thoughts, guessing a person’s weight, when he knows you’re doing so, introduces possibilities for error, and it is fortunate that this example was not tried out by James Surowiecki, or he might not have bothered with his study.

I like to think the scale was broken. 🙂

Question 4. Correct answer: kitten heels.

Type of heel Number percent (%) Status
1. Stiletto 6 40% wrong
2. Pompadour 4 26% wrong
3. Kitten 3 20% correct
4. Achilles 0 0% wrong
5. No response 2 13% N/A

What was annoying was the refusal of two people to guess. This creates an element of uncertainty about the results: did they know stiletto was wrong, or think it was “too obvious”? If so, there’s a 50% chance that kitten would have got an extra vote.

This was as close to a “Who Wants to be a Millionaire” ask the audience question as I could design. The presentation of the options was laid out in the same way as they are on the television screen. The order was chosen randomly.

Considering how far from a contemporary fashion-conscious crowd my audience was, 20% picking the right answer is more than I would have expected. Equally good: no wrong answer got a majority, which would help the crowd come to the conclusion that this time, they might need an expert.

Question 5. Who do you think will win the election?
Conservatives: 10 (67%)
No majority: 5 (33%)
Others: 0

I suspect the correct odds were precisely correct at the time of this questionnaire (11th January 2010).

The moral is that asking people what they think will happen is more accurate than asking what they want to happen.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: