IDEAS home Printed from
   My bibliography  Save this article

The relationship between crowd majority and accuracy for binary decisions


  • Michael D. Lee
  • Megan N. Lee


We consider the wisdom of the crowd situation in which individuals make binary decisions, and the majority answer is used as the group decision. Using data sets from nine different domains, we examine the relationship between the size of the majority and the accuracy of the crowd decisions. We find empirically that these calibration curves take many different forms for different domains, and the distribution of majority sizes over decisions in a domain also varies widely. We develop a growth model for inferring and interpreting the calibration curve in a domain, and apply it to the same nine data sets using Bayesian methods. The modeling approach is able to infer important qualitative properties of a domain, such as whether it involves decisions that have ground truths or are inherently uncertain. It is also able to make inferences about important quantitative properties of a domain, such as how quickly the crowd accuracy increases as the size of the majority increases. We discuss potential applications of the measurement model, and the need to develop a psychological account of the variety of calibration curves that evidently exist.

Suggested Citation

  • Michael D. Lee & Megan N. Lee, 2017. "The relationship between crowd majority and accuracy for binary decisions," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(4), pages 328-343, July.
  • Handle: RePEc:jdm:journl:v:12:y:2017:i:4:p:328-343

    Download full text from publisher

    File URL:
    Download Restriction: no

    File URL:
    Download Restriction: no

    References listed on IDEAS

    1. Ladha, Krishna K., 1995. "Information pooling through majority-rule voting: Condorcet's jury theorem with correlated votes," Journal of Economic Behavior & Organization, Elsevier, vol. 26(3), pages 353-372, May.
    2. Rachel Croson & James Sundali, 2005. "The Gambler’s Fallacy and the Hot Hand: Empirical Data from Casinos," Journal of Risk and Uncertainty, Springer, vol. 30(3), pages 195-209, May.
    3. Joseph P. Simmons & Leif D. Nelson & Jeff Galak & Shane Frederick, 2011. "Intuitive Biases in Choice versus Estimation: Implications for the Wisdom of Crowds," Journal of Consumer Research, Oxford University Press, vol. 38(1), pages 1-15.
    4. Michael D. Lee & Irina Danileiko, 2014. "Using cognitive models to combine probability estimates," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 9(3), pages 259-273, May.
    5. Blinder, Alan S & Morgan, John, 2005. "Are Two Heads Better than One? Monetary Policy by Committee," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 37(5), pages 789-811, October.
    6. Clintin P. Davis-Stober & David V. Budescu & Stephen B. Broomell & Jason Dana, 2015. "The Composition of Optimally Wise Crowds," Decision Analysis, INFORMS, vol. 12(3), pages 130-143.
    7. Lionel Page & Robert T. Clemen, 2013. "Do Prediction Markets Produce Well‐Calibrated Probability Forecasts?-super-," Economic Journal, Royal Economic Society, vol. 123(568), pages 491-513, May.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Michael D. Lee & Irina Danileiko & Julie Vi, 2018. "Testing the ability of the surprisingly popular method to predict NFL games," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(4), pages 322-333, July.


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jdm:journl:v:12:y:2017:i:4:p:328-343. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jonathan Baron). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.