IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v248y2016i3p1021-1030.html
   My bibliography  Save this article

Probabilistic forecasting with discrete choice models: Evaluating predictions with pseudo-coefficients of determination

Author

Listed:
  • Sung, Ming-Chien
  • McDonald, David C.J.
  • Johnson, Johnnie E.V.

Abstract

Probabilistic forecasts from discrete choice models, which are widely used in marketing science and competitive event forecasting, are often best evaluated out-of-sample using pseudo-coefficients of determination, or pseudo-R2s. However, there is a danger of misjudging the accuracy of forecast probabilities of event outcomes, based on observed frequencies, because of issues related to pseudo-R2s. First, we show that McFadden’s pseudo-R2 varies predictably with the number of alternatives in the choice set. Then we evaluate the relative merits of two methods (bootstrap and asymptotic) for estimating the variance of pseudo-R2s so that their values can be appropriately compared across non-nested models. Finally, in the context of competitive event forecasting, where the accuracy of forecasts has direct economic consequence, we derive new R2 measures that can be used to assess the economic value of forecasts. Throughout, we illustrate using data drawn from UK and Ireland horse race betting markets.

Suggested Citation

  • Sung, Ming-Chien & McDonald, David C.J. & Johnson, Johnnie E.V., 2016. "Probabilistic forecasting with discrete choice models: Evaluating predictions with pseudo-coefficients of determination," European Journal of Operational Research, Elsevier, vol. 248(3), pages 1021-1030.
  • Handle: RePEc:eee:ejores:v:248:y:2016:i:3:p:1021-1030
    DOI: 10.1016/j.ejor.2015.08.068
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0377221715008280
    Download Restriction: Full text for ScienceDirect subscribers only

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Schnytzer, Adi & Lamers, Martien & Makropoulou, Vasiliki, 2010. "The impact of insider trading on forecasting in a bookmakers' horse betting market," International Journal of Forecasting, Elsevier, vol. 26(3), pages 537-542, July.
    2. Shaoming Cheng & Roger Stough, 2006. "Location decisions of Japanese new manufacturing plants in China: a discrete-choice analysis," The Annals of Regional Science, Springer;Western Regional Science Association, vol. 40(2), pages 369-387, June.
    3. Wolfers, Justin & Zitzewitz, Eric, 2006. "Five Open Questions About Prediction Markets," CEPR Discussion Papers 5562, C.E.P.R. Discussion Papers.
    4. D. J. Johnstone, 2011. "Economic Interpretation of Probabilities Estimated by Maximum Likelihood or Score," Management Science, INFORMS, vol. 57(2), pages 308-314, February.
    5. Franck, Egon & Verbeek, Erwin & Nüesch, Stephan, 2010. "Prediction accuracy of different market structures -- bookmakers versus a betting exchange," International Journal of Forecasting, Elsevier, vol. 26(3), pages 448-459, July.
    6. Veall, Michael R & Zimmermann, Klaus F, 1996. " Pseudo-R-[superscript 2] Measures for Some Common Limited Dependent Variable Models," Journal of Economic Surveys, Wiley Blackwell, vol. 10(3), pages 241-259, September.
    7. Abe, Makoto, 1999. "A Generalized Additive Model for Discrete-Choice Data," Journal of Business & Economic Statistics, American Statistical Association, vol. 17(3), pages 271-284, July.
    8. Lessmann, Stefan & Sung, Ming-Chien & Johnson, Johnnie E.V. & Ma, Tiejun, 2012. "A new methodology for generating and combining statistical forecasting models to enhance competitive event prediction," European Journal of Operational Research, Elsevier, vol. 218(1), pages 163-174.
    9. Liu, Yu-Hsin, 2011. "Incorporating scatter search and threshold accepting in finding maximum likelihood estimates for the multinomial probit model," European Journal of Operational Research, Elsevier, vol. 211(1), pages 130-138, May.
    10. Lessmann, Stefan & Sung, Ming-Chien & Johnson, Johnnie E.V., 2009. "Identifying winners of competitive events: A SVM-based classification model for horserace prediction," European Journal of Operational Research, Elsevier, vol. 196(2), pages 569-577, July.
    11. Press, S. James & Zellner, Arnold, 1978. "Posterior distribution for the multiple correlation coefficient with fixed regressors," Journal of Econometrics, Elsevier, vol. 8(3), pages 307-321, December.
    12. Ming-Chien Sung & Johnnie E. V. Johnson & John Peirson, 2012. "Discovering a Profitable Trading Strategy in an Apparently Efficient Market: Exploiting the Actions of Less Informed Traders in Speculative Markets," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 39(7-8), pages 1131-1159, September.
    13. Smith, Michael A. & Vaughan Williams, Leighton, 2010. "Forecasting horse race outcomes: New evidence on odds bias in UK betting markets," International Journal of Forecasting, Elsevier, vol. 26(3), pages 543-550, July.
    14. Hyndman, Rob J. & Koehler, Anne B., 2006. "Another look at measures of forecast accuracy," International Journal of Forecasting, Elsevier, vol. 22(4), pages 679-688.
    15. Lin, Kyle Y. & Sibdari, Soheil Y., 2009. "Dynamic price competition with discrete customer choices," European Journal of Operational Research, Elsevier, vol. 197(3), pages 969-980, September.
    16. Ohtani, Kazuhiro, 2000. "Bootstrapping R2 and adjusted R2 in regression analysis," Economic Modelling, Elsevier, vol. 17(4), pages 473-483, December.
    Full references (including those not matched with items on IDEAS)

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:248:y:2016:i:3:p:1021-1030. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Dana Niculescu). General contact details of provider: http://www.elsevier.com/locate/eor .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.