IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v50y2004i5p597-604.html
   My bibliography  Save this article

Probability Judgments for Continuous Quantities: Linear Combinations and Calibration

Author

Listed:
  • Stephen C. Hora

    (Department of Business Administration, University of Hawaii-Hilo, Hilo, Hawaii 96720.)

Abstract

Expert judgment elicitation is often required in probabilistic decision making and the evaluation of risk. One measure of the quality of probability distributions given by experts is calibration--the faithfulness of the probabilities in an empirically verifiable sense. A method of measuring calibration for continuous probability distributions is presented here. A discussion of the impact of using linear rules for combining such judgments is given and an empirical demonstration is given using data collected from experts participating in a large-scale risk study. It is shown by theoretical argument that combining well-calibrated distributions of individual experts using linear rules can only result in reducing calibration. In contrast, it is demonstrated, both by example and empirically, that an equally weighted linear combination of experts who tend to be "overconfident" can produce distributions that are better calibrated than the experts' individual distributions. Using data from training exercises, it is shown that the improvement in calibration is rapid as the number of experts is increased from one to five or six, but there is only modest improvement from increasing the number of experts beyond that point.

Suggested Citation

  • Stephen C. Hora, 2004. "Probability Judgments for Continuous Quantities: Linear Combinations and Calibration," Management Science, INFORMS, vol. 50(5), pages 597-604, May.
  • Handle: RePEc:inm:ormnsc:v:50:y:2004:i:5:p:597-604
    DOI: 10.1287/mnsc.1040.0205
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.1040.0205
    Download Restriction: no

    File URL: https://libkey.io/10.1287/mnsc.1040.0205?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    2. Hora, Stephen C. & Hora, Judith A. & Dodd, Nancy G., 1992. "Assessment of probability distributions for continuous random variables: A comparison of the bisection and fixed value methods," Organizational Behavior and Human Decision Processes, Elsevier, vol. 51(1), pages 133-155, February.
    3. Robert L. Winkler, 1968. "The Consensus of Subjective Probability Distributions," Management Science, INFORMS, vol. 15(2), pages 61-75, October.
    4. Peter A. Morris, 1974. "Decision Analysis Expert Use," Management Science, INFORMS, vol. 20(9), pages 1233-1241, May.
    5. Peter A. Morris, 1977. "Combining Expert Judgments: A Bayesian Approach," Management Science, INFORMS, vol. 23(7), pages 679-693, March.
    6. Juslin, Peter, 1994. "The Overconfidence Phenomenon as a Consequence of Informal Experimenter-Guided Selection of Almanac Items," Organizational Behavior and Human Decision Processes, Elsevier, vol. 57(2), pages 226-246, February.
    7. Brenner, Lyle A. & Koehler, Derek J. & Liberman, Varda & Tversky, Amos, 1996. "Overconfidence in Probability and Frequency Judgments: A Critical Examination," Organizational Behavior and Human Decision Processes, Elsevier, vol. 65(3), pages 212-219, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    2. Christian Kascha & Francesco Ravazzolo, 2010. "Combining inflation density forecasts," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 29(1-2), pages 231-250.
    3. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    4. Neil A. Stiber & Mitchell J. Small & Marina Pantazidou, 2004. "Site‐Specific Updating and Aggregation of Bayesian Belief Network Models for Multiple Experts," Risk Analysis, John Wiley & Sons, vol. 24(6), pages 1529-1538, December.
    5. David M. Pennock & Michael P. Wellman, 2005. "Graphical Models for Groups: Belief Aggregation and Risk Sharing," Decision Analysis, INFORMS, vol. 2(3), pages 148-164, September.
    6. Li, Yongquan & Zhu, Kaijie, 2009. "Information acquisition in new product introduction," European Journal of Operational Research, Elsevier, vol. 198(2), pages 618-625, October.
    7. Fu, Qi & Zhu, Kaijie, 2010. "Endogenous information acquisition in supply chain management," European Journal of Operational Research, Elsevier, vol. 201(2), pages 454-462, March.
    8. Wang, W., 1997. "Subjective estimation of the delay time distribution in maintenance modelling," European Journal of Operational Research, Elsevier, vol. 99(3), pages 516-529, June.
    9. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    10. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    11. Hall, Stephen G. & Mitchell, James, 2007. "Combining density forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 1-13.
    12. Ali Mosleh & George Apostolakis, 1986. "The Assessment of Probability Distributions from Expert Opinions with an Application to Seismic Fragility Curves," Risk Analysis, John Wiley & Sons, vol. 6(4), pages 447-461, December.
    13. Thibault Gajdos & Jean-Christophe Vergnaud, 2013. "Decisions with conflicting and imprecise information," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 41(2), pages 427-452, July.
    14. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    15. Enrique Moral-Benito, 2015. "Model Averaging In Economics: An Overview," Journal of Economic Surveys, Wiley Blackwell, vol. 29(1), pages 46-75, February.
    16. Anne Sofie Jore & James Mitchell & Shaun P. Vahey, 2010. "Combining forecast densities from VARs with uncertain instabilities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 621-634.
    17. Tommaso Proietti & Martyna Marczak & Gianluigi Mazzi, 2017. "Euromind‐ D : A Density Estimate of Monthly Gross Domestic Product for the Euro Area," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(3), pages 683-703, April.
    18. Garratt, Anthony & Mise, Emi, 2014. "Forecasting exchange rates using panel model and model averaging," Economic Modelling, Elsevier, vol. 37(C), pages 32-40.
    19. Yann Braouezec, 2010. "Committee, Expert Advice, and the Weighted Majority Algorithm: An Application to the Pricing Decision of a Monopolist," Computational Economics, Springer;Society for Computational Economics, vol. 35(3), pages 245-267, March.
    20. Stephen Hora & Erim Kardeş, 2015. "Calibration, sharpness and the weighting of experts in a linear opinion pool," Annals of Operations Research, Springer, vol. 229(1), pages 429-450, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:50:y:2004:i:5:p:597-604. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.