IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v58y2010i5p1440-1449.html
   My bibliography  Save this article

An Analytic Method for Evaluating the Performance of Aggregation Rules for Probability Densities

Author

Listed:
  • Stephen C. Hora

    (Viterbi School of Engineering, and School of Policy, Planning, and Development, University of Southern California, Los Angeles, California 90089)

Abstract

It is shown how infinite sequences of densities with defined properties can be used to evaluate the expected performance of mathematical aggregation rules for elicited densities. The performance of these rules is measured through the average variance, calibration, and average Brier score of the aggregates. A general result for the calibration of the arithmetic average of densities from well-calibrated independent experts is given. Arithmetic and geometric aggregation rules are compared using sequences of normal densities. Sequences are developed that exhibit dependence among experts and lack of calibration. The impact of correlation, number of experts, and degree of calibration on the performance of the aggregation is demonstrated.

Suggested Citation

  • Stephen C. Hora, 2010. "An Analytic Method for Evaluating the Performance of Aggregation Rules for Probability Densities," Operations Research, INFORMS, vol. 58(5), pages 1440-1449, October.
    • Handle: RePEc:inm:oropre:v:58:y:2010:i:5:p:1440-1449
    DOI: 10.1287/opre.1100.0789
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/opre.1100.0789
    Download Restriction: no
    File URL: https://libkey.io/10.1287/opre.1100.0789?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Allan H. Murphy & Robert L. Winkler, 1977. "Reliability of Subjective Probability Forecasts of Precipitation and Temperature," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 26(1), pages 41-47, March.
    2. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    3. Stephen C. Hora, 2004. "Probability Judgments for Continuous Quantities: Linear Combinations and Calibration," Management Science, INFORMS, vol. 50(5), pages 597-604, May.
    4. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    5. Dean Croushore, 1993. "Introducing: the survey of professional forecasters," Business Review, Federal Reserve Bank of Philadelphia, issue Nov, pages 3-15.
    6. Robert F. Bordley & Ronald W. Wolff, 1981. "Note---On the Aggregation of Individual Probability Estimates," Management Science, INFORMS, vol. 27(8), pages 959-964, August.
    7. Robert T. Clemen & Terence Reilly, 1999. "Correlations and Copulas for Decision and Risk Analysis," Management Science, INFORMS, vol. 45(2), pages 208-224, February.
    8. Robert T. Clemen & Robert L. Winkler, 1985. "Limits for the Precision and Value of Information from Dependent Sources," Operations Research, INFORMS, vol. 33(2), pages 427-442, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. James K. Hammitt & Yifan Zhang, 2013. "Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
    2. Kenneth C. Lichtendahl & Yael Grushka-Cockayne & Robert L. Winkler, 2013. "Is It Better to Average Probabilities or Quantiles?," Management Science, INFORMS, vol. 59(7), pages 1594-1611, July.
    3. Roberto Casarin & Giulia Mantoan & Francesco Ravazzolo, 2016. "Bayesian Calibration of Generalized Pools of Predictive Distributions," Econometrics, MDPI, vol. 4(1), pages 1-24, March.
    4. Stephen C. Hora, 2013. "A Note on the Aggregation of Event Probabilities," Risk Analysis, John Wiley & Sons, vol. 33(5), pages 909-914, May.
    5. Chen Wang & Vicki M. Bier, 2013. "Expert Elicitation of Adversary Preferences Using Ordinal Judgments," Operations Research, INFORMS, vol. 61(2), pages 372-385, April.
    6. Stephen C. Hora & Benjamin R. Fransen & Natasha Hawkins & Irving Susel, 2013. "Median Aggregation of Distribution Functions," Decision Analysis, INFORMS, vol. 10(4), pages 279-291, December.
    7. Mitchell, James, 2013. "The Recalibrated and Copula Opinion Pools," EMF Research Papers 02, Economic Modelling and Forecasting Group.
    8. Samrat Chatterjee & Stephen C. Hora & Heather Rosoff, 2015. "Portfolio Analysis of Layered Security Measures," Risk Analysis, John Wiley & Sons, vol. 35(3), pages 459-475, March.
    9. Liu, Peng & Qiu, Yongping & Hu, Juntao & Tong, Jiejuan & Zhao, Jun & Li, Zhizhong, 2020. "Expert judgments for performance shaping Factors’ multiplier design in human reliability analysis," Reliability Engineering and System Safety, Elsevier, vol. 194(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    2. Stephen C. Hora, 2013. "A Note on the Aggregation of Event Probabilities," Risk Analysis, John Wiley & Sons, vol. 33(5), pages 909-914, May.
    3. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    4. James K. Hammitt & Yifan Zhang, 2013. "Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
    5. Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.
    6. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    7. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    8. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    9. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    10. A. E. Ades & Karl Claxton & Mark Sculpher, 2006. "Evidence synthesis, parameter correlation and probabilistic sensitivity analysis," Health Economics, John Wiley & Sons, Ltd., vol. 15(4), pages 373-381, April.
    11. Jaspersen, Johannes G., 2022. "Convex combinations in judgment aggregation," European Journal of Operational Research, Elsevier, vol. 299(2), pages 780-794.
    12. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    13. Mark Freeman & Ben Groom, 2015. "Using equity premium survey data to estimate future wealth," Review of Quantitative Finance and Accounting, Springer, vol. 45(4), pages 665-693, November.
    14. Song, Haiyan & Wen, Long & Liu, Chang, 2019. "Density tourism demand forecasting revisited," Annals of Tourism Research, Elsevier, vol. 75(C), pages 379-392.
    15. Paola Monari & Patrizia Agati, 2001. "Fiducial inference in combining expert judgements," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 10(1), pages 81-97, January.
    16. Jason R. W. Merrick, 2008. "Getting the Right Mix of Experts," Decision Analysis, INFORMS, vol. 5(1), pages 43-52, March.
    17. Bian, Wenliang & Shang, Jennifer & Zhang, Juliang, 2016. "Two-way information sharing under supply chain competition," International Journal of Production Economics, Elsevier, vol. 178(C), pages 82-94.
    18. Stephen C. Hora & Benjamin R. Fransen & Natasha Hawkins & Irving Susel, 2013. "Median Aggregation of Distribution Functions," Decision Analysis, INFORMS, vol. 10(4), pages 279-291, December.
    19. James S. Dyer & James E. Smith, 2021. "Innovations in the Science and Practice of Decision Analysis: The Role of Management Science," Management Science, INFORMS, vol. 67(9), pages 5364-5378, September.
    20. Stephen Hora & Erim Kardeş, 2015. "Calibration, sharpness and the weighting of experts in a linear opinion pool," Annals of Operations Research, Springer, vol. 229(1), pages 429-450, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:58:y:2010:i:5:p:1440-1449. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.