IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v11y2014i2p133-145.html

Two Reasons to Make Aggregated Probability Forecasts More Extreme

Author

Listed:
  • Jonathan Baron

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Barbara A. Mellers

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Philip E. Tetlock

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Eric Stone

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Lyle H. Ungar

    (Department of Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

Abstract

When aggregating the probability estimates of many individuals to form a consensus probability estimate of an uncertain future event, it is common to combine them using a simple weighted average. Such aggregated probabilities correspond more closely to the real world if they are transformed by pushing them closer to 0 or 1. We explain the need for such transformations in terms of two distorting factors: The first factor is the compression of the probability scale at the two ends, so that random error tends to push the average probability toward 0.5. This effect does not occur for the median forecast, or, arguably, for the mean of the log odds of individual forecasts. The second factor---which affects mean, median, and mean of log odds---is the result of forecasters taking into account their individual ignorance of the total body of information available. Individual confidence in the direction of a probability judgment (high/low) thus fails to take into account the wisdom of crowds that results from combining different evidence available to different judges. We show that the same transformation function can approximately eliminate both distorting effects with different parameters for the mean and the median. And we show how, in principle, use of the median can help distinguish the two effects.

Suggested Citation

  • Jonathan Baron & Barbara A. Mellers & Philip E. Tetlock & Eric Stone & Lyle H. Ungar, 2014. "Two Reasons to Make Aggregated Probability Forecasts More Extreme," Decision Analysis, INFORMS, vol. 11(2), pages 133-145, June.
    • Handle: RePEc:inm:ordeca:v:11:y:2014:i:2:p:133-145
    DOI: 10.1287/deca.2014.0293
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/deca.2014.0293
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.2014.0293?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Wallsten, Thomas S. & Diederich, Adele, 2001. "Understanding pooled subjective probability estimates," Mathematical Social Sciences, Elsevier, vol. 41(1), pages 1-18, January.
    2. Satopää, Ville A. & Baron, Jonathan & Foster, Dean P. & Mellers, Barbara A. & Tetlock, Philip E. & Ungar, Lyle H., 2014. "Combining multiple probability predictions using a simple logit model," International Journal of Forecasting, Elsevier, vol. 30(2), pages 344-356.
    3. David J. Johnstone & Victor Richmond R. Jose & Robert L. Winkler, 2011. "Tailored Scoring Rules for Probabilities," Decision Analysis, INFORMS, vol. 8(4), pages 256-268, December.
    4. A. Drăgulescu & V.M. Yakovenko, 2001. "Evidence for the exponential distribution of income in the USA," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 20(4), pages 585-589, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jason Dana & Pavel Atanasov & Philip Tetlock & Barbara Mellers, 2019. "Are markets more accurate than polls? The surprising informational value of “just askingâ€," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(2), pages 135-147, March.
    2. Marcellin Martinie & Tom Wilkening & Piers D L Howe, 2020. "Using meta-predictions to identify experts in the crowd when past performance is unknown," PLOS ONE, Public Library of Science, vol. 15(4), pages 1-11, April.
    3. Ville A. Satopää & Robin Pemantle & Lyle H. Ungar, 2016. "Modeling Probability Forecasts via Information Diversity," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1623-1633, October.
    4. David R. Mandel & Christopher W. Karvetski & Mandeep K. Dhami, 2018. "Boosting intelligence analysts’ judgment accuracy: What works, what fails?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(6), pages 607-621, November.
    5. Karimi Motahhar, Vahid & Gruca, Thomas S., 2025. "How does training improve individual forecasts? Modeling differences in compensatory and non-compensatory biases in geopolitical forecasts," International Journal of Forecasting, Elsevier, vol. 41(2), pages 487-498.
    6. Satopää, Ville A., 2021. "Improving the wisdom of crowds with analysis of variance of predictions of related outcomes," International Journal of Forecasting, Elsevier, vol. 37(4), pages 1728-1747.
    7. Huck, Nicolas, 2019. "Large data sets and machine learning: Applications to statistical arbitrage," European Journal of Operational Research, Elsevier, vol. 278(1), pages 330-342.
    8. Don A. Moore & Samuel A. Swift & Angela Minster & Barbara Mellers & Lyle Ungar & Philip Tetlock & Heather H. J. Yang & Elizabeth R. Tenney, 2017. "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," Management Science, INFORMS, vol. 63(11), pages 3552-3565, November.
    9. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    10. Robert Mislavsky & Celia Gaertig, 2022. "Combining Probability Forecasts: 60% and 60% Is 60%, but Likely and Likely Is Very Likely," Management Science, INFORMS, vol. 68(1), pages 541-563, January.
    11. Satopää, Ville A. & Salikhov, Marat & Tetlock, Philip E. & Mellers, Barbara, 2023. "Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model," International Journal of Forecasting, Elsevier, vol. 39(1), pages 470-485.
    12. Tao Lin & Yiling Chen, 2022. "Sample Complexity of Forecast Aggregation," Papers 2207.13126, arXiv.org, revised Oct 2023.
    13. Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    14. Peker, Cem & Wilkening, Tom, 2025. "Robust recalibration of aggregate probability forecasts using meta-beliefs," International Journal of Forecasting, Elsevier, vol. 41(2), pages 613-630.
    15. Asa B. Palley & Jack B. Soll, 2019. "Extracting the Wisdom of Crowds When Information Is Shared," Management Science, INFORMS, vol. 67(5), pages 2291-2309, May.
    16. Andrew Gelman & Jessica Hullman & Christopher Wlezien & George Elliott Morris, 2020. "Information, incentives, and goals in election forecasts," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 863-880, September.
    17. Ying Han & David V. Budescu, 2022. "Recalibrating probabilistic forecasts to improve their accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 17(1), pages 91-123, January.
    18. Michael D. Lee & Irina Danileiko, 2014. "Using cognitive models to combine probability estimates," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 9(3), pages 259-273, May.
    19. Joshua D. Kertzer, 2017. "Microfoundations in international relations," Conflict Management and Peace Science, Peace Science Society (International), vol. 34(1), pages 81-97, January.
    20. Karvetski, Christopher W. & Meinel, Carolyn & Maxwell, Daniel T. & Lu, Yunzi & Mellers, Barbara A. & Tetlock, Philip E., 2022. "What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?," International Journal of Forecasting, Elsevier, vol. 38(2), pages 688-704.
    21. Jared A. Beekman & Ronald F. A. Woodaman & Dennis M. Buede, 2020. "A Review of Probabilistic Opinion Pooling Algorithms with Application to Insider Threat Detection," Decision Analysis, INFORMS, vol. 17(1), pages 39-55, March.
    22. Anca M. Hanea & Marissa F. McBride & Mark A. Burgman & Bonnie C. Wintle, 2018. "The Value of Performance Weights and Discussion in Aggregated Expert Judgments," Risk Analysis, John Wiley & Sons, vol. 38(9), pages 1781-1794, September.
    23. Spyros Galanis & Sergei Mikhalishchev, 2024. "Information Aggregation with Costly Information Acquisition," Papers 2406.07186, arXiv.org, revised Jun 2025.
    24. David R. Mandel & Daniel Irwin, 2021. "Tracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study," Futures & Foresight Science, John Wiley & Sons, vol. 3(3-4), September.
    25. Ying Han & David Budescu, 2019. "A universal method for evaluating the quality of aggregators," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 395-411, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rashkovskiy, Sergey A., 2019. "‘Bosons’ and ‘fermions’ in social and economic systems," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 514(C), pages 90-104.
    2. Ellis Scharfenaker, Markus P.A. Schneider, 2019. "Labor Market Segmentation and the Distribution of Income: New Evidence from Internal Census Bureau Data," Working Paper Series, Department of Economics, University of Utah 2019_08, University of Utah, Department of Economics.
    3. Coelho, Ricardo & Richmond, Peter & Barry, Joseph & Hutzler, Stefan, 2008. "Double power laws in income and wealth distributions," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 387(15), pages 3847-3851.
    4. Anwar Shaikh & Amr Ragab, 2009. "Ingresos de la Gran Mayoría (IGM): Una Nueva Medida de la Desigualdad Mundial," Policy Research Brief 7, International Policy Centre.
    5. Karimi, Majid & Zaerpour, Nima, 2022. "Put your money where your forecast is: Supply chain collaborative forecasting with cost-function-based prediction markets," European Journal of Operational Research, Elsevier, vol. 300(3), pages 1035-1049.
    6. Costas Efthimiou & Adam Wearne, 2016. "Household Income Distribution in the USA," Papers 1602.06234, arXiv.org.
    7. Anirban Chakraborti & Ioane Muni Toke & Marco Patriarca & Frédéric Abergel, 2011. "Econophysics review: II. Agent-based models," Post-Print hal-00621059, HAL.
    8. repec:osf:metaar:rxmh7_v1 is not listed on IDEAS
    9. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    10. Hernández-Ramírez, E. & del Castillo-Mussot, M. & Hernández-Casildo, J., 2021. "World per capita gross domestic product measured nominally and across countries with purchasing power parity: Stretched exponential or Boltzmann–Gibbs distribution?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 568(C).
    11. Rakesh K. Sarin, 2013. "From the Editor ---Median Aggregation, Scoring Rules, Expert Forecasts, Choices with Binary Attributes, Portfolio with Dependent Projects, and Information Security," Decision Analysis, INFORMS, vol. 10(4), pages 277-278, December.
    12. Makowsky, Michael, 2009. "Religious Extremism, Clubs, and Civil Liberties: A Model of Religious Populations," MPRA Paper 14358, University Library of Munich, Germany.
    13. Pengfei Han & Zhu Wang, 2021. "Technology Adoption and Leapfrogging: Racing for Mobile Payments," Working Paper 21-05, Federal Reserve Bank of Richmond.
    14. Callealta Barroso, Francisco Javier & García-Pérez, Carmelo & Prieto-Alaiz, Mercedes, 2020. "Modelling income distribution using the log Student’s t distribution: New evidence for European Union countries," Economic Modelling, Elsevier, vol. 89(C), pages 512-522.
    15. Néda, Zoltán & Gere, István & Biró, Tamás S. & Tóth, Géza & Derzsy, Noemi, 2020. "Scaling in income inequalities and its dynamical origin," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 549(C).
    16. Hernández-Pérez, R., 2010. "An analogy of the size distribution of business firms with Bose–Einstein statistics," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 389(18), pages 3837-3843.
    17. Christophe Schinckus & Çınla Akdere, 2015. "Towards a New Way of Teaching Statistics in Economics: The Case for Econophysics," Ekonomi-tek - International Economics Journal, Turkish Economic Association, vol. 4(3), pages 89-108, September.
    18. Behrens, Kristian & Murata, Yasusada, 2012. "Globalization and individual gains from trade," Journal of Monetary Economics, Elsevier, vol. 59(8), pages 703-720.
    19. Satopää, Ville A., 2021. "Improving the wisdom of crowds with analysis of variance of predictions of related outcomes," International Journal of Forecasting, Elsevier, vol. 37(4), pages 1728-1747.
    20. Peyman Mahmoudi & Alireza Ghaemi & Allahbakhsh Rigi & Seyed Mahdi Amir Jahanshahi, 2021. "RETRACTED ARTICLE: Recommendations for modifying the Standardized Precipitation Index (SPI) for Drought Monitoring in Arid and Semi-arid Regions," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 35(10), pages 3253-3275, August.
    21. Ricardo Lopez-Ruiz, 2012. "Complex Systems with Trivial Dynamics," Papers 1210.6481, arXiv.org.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:11:y:2014:i:2:p:133-145. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.