Related Work: Measuring the Crowd Within

Measuring the Crowd Within: Probabilistic Representations Within Individuals
By Edward Vul and Harold Pashler
Psychological Science, 19(7): 2008.

Crowdsourcing researchers are familiar with the idea of the wisdom-of-crowds, whereby information from multiple different people can be pooled to produce a better outcome than any individual could produce alone. Vul and Pashler’s paper provides some fun common examples of wisdom-of-crowds in action, including:

• A group of people can guess the weight of an ox better than an individual.
• Polling the audience on Who Wants to be a Millionaire is successful.
• The combination of multiple financial forecasts better than individual forecasts.

This paper builds on the notion of the wisdom-of-crowds to ask: Can information collected multiple times from the same individual can also improve the outcome? The authors study this by asking participants to answer questions like, “What percentage of the world’s airports are in the United States?” twice. They find that when they average both answers they get a more accurate response than if they were to look at either answer individually.

However, averaging two answers from the same individual isn’t nearly as good as averaging two answers from two different participants. In fact, they find that “you can gain about 1/10th as much from asking yourself the same question twice as you can from getting a second opinion from someone else.” When discussing the wisdom-of-crowds, Vul and Pashler note that averaging “is be more accurate as long as some of the error of one individual is statistically independent of the error of other individuals.” In the case where the samples are drawn from the same person, they are clearly not independent. This finding is consistent with work by Ariely et al. that found that averaging multiple guesses from an individual do not help as much as multiple guesses from different people “because of greater violations of pairwise independence.”

It may be possible to introduce greater independence in the way a person answers the same question by being thoughtful about how the samples are collected. When Vul and Pashler made participants wait three weeks between providing answers, they found that “the benefit of reasking yourself the same question rises to 1/3 the value of a second opinion.” Considering alternative or opposite outcomes has been shown to help people overcome anchoring effects and produce less biased judgments. This suggests it may also be possible to help introduce individual variation by, for example, showing them different perspectives (see talk by Mark Styvers, slides 49-53).

Summary: Collecting information multiple times from the same individual can improve the outcome, but not as much as if the information were collected from multiple different people instead. Individual outcomes can be improved by encouraging a person to give more independent responses.

Question: I am intrigued by the fact that people’s second guesses were significantly worse than their first guesses, regardless of whether there was a delay in the response or not. Any idea why that might be the case?

Edited to add a pointer from Hal Pashler in response to this post: The Wisdom of Many in One Mind