IDEAS home Printed from https://ideas.repec.org/a/bpj/jqsprt/v8y2012i1n12.html
   My bibliography  Save this article

Solving the Problem of Inadequate Scoring Rules for Assessing Probabilistic Football Forecast Models

Author

Listed:
  • Constantinou Anthony Costa

    (Queen Mary, University of London)

  • Fenton Norman Elliott

    (Queen Mary, University of London)

Abstract

Despite the massive popularity of probabilistic (association) football forecasting models, and the relative simplicity of the outcome of such forecasts (they require only three probability values corresponding to home win, draw, and away win) there is no agreed scoring rule to determine their forecast accuracy. Moreover, the various scoring rules used for validation in previous studies are inadequate since they fail to recognise that football outcomes represent a ranked (ordinal) scale. This raises severe concerns about the validity of conclusions from previous studies. There is a well-established generic scoring rule, the Rank Probability Score (RPS), which has been missed by previous researchers, but which properly assesses football forecasting models.

Suggested Citation

  • Constantinou Anthony Costa & Fenton Norman Elliott, 2012. "Solving the Problem of Inadequate Scoring Rules for Assessing Probabilistic Football Forecast Models," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 8(1), pages 1-14, March.
    • Handle: RePEc:bpj:jqsprt:v:8:y:2012:i:1:n:12
    DOI: 10.1515/1559-0410.1418
    as

    Download full text from publisher

    File URL: https://doi.org/10.1515/1559-0410.1418
    Download Restriction: For access to full text, subscription to the journal or payment for the individual article is required.
    File URL: https://libkey.io/10.1515/1559-0410.1418?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Forrest, David & Goddard, John & Simmons, Robert, 2005. "Odds-setters as forecasters: The case of English football," International Journal of Forecasting, Elsevier, vol. 21(3), pages 551-564.
    2. Victor Richmond R. Jose & Robert F. Nau & Robert L. Winkler, 2009. "Sensitivity to Distance and Baseline Distributions in Forecast Evaluation," Management Science, INFORMS, vol. 55(4), pages 582-590, April.
    3. Fildes, Robert & Stekler, Herman, 2002. "The state of macroeconomic forecasting," Journal of Macroeconomics, Elsevier, vol. 24(4), pages 435-468, December.
    4. Fildes, Robert & Stekler, Herman, 2002. "Reply to the comments on 'The state of macroeconomic forecasting'," Journal of Macroeconomics, Elsevier, vol. 24(4), pages 503-505, December.
    5. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    6. Garthwaite, Paul H. & Kadane, Joseph B. & O'Hagan, Anthony, 2005. "Statistical Methods for Eliciting Probability Distributions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 680-701, June.
    7. I. Graham & H. Stott, 2008. "Predicting bookmaker odds and efficiency for UK football," Applied Economics, Taylor & Francis Journals, vol. 40(1), pages 99-109.
    8. Gianluca Baio & Marta Blangiardo, 2010. "Bayesian hierarchical model for the prediction of football results," Journal of Applied Statistics, Taylor & Francis Journals, vol. 37(2), pages 253-264.
    9. Armstrong, J. Scott & Collopy, Fred, 1992. "Error measures for generalizing about forecasting methods: Empirical comparisons," International Journal of Forecasting, Elsevier, vol. 8(1), pages 69-80, June.
    10. Dixon, Mark J. & Pope, Peter F., 2004. "The value of statistical forecasts in the UK association football betting market," International Journal of Forecasting, Elsevier, vol. 20(4), pages 697-711.
    11. Leitch, Gordon & Tanner, J Ernest, 1991. "Economic Forecast Evaluation: Profits versus the Conventional Error Measures," American Economic Review, American Economic Association, vol. 81(3), pages 580-590, June.
    12. Hendry, David F, 1997. "The Econometrics of Macroeconomic Forecasting," Economic Journal, Royal Economic Society, vol. 107(444), pages 1330-1357, September.
    13. Goddard, John, 2005. "Regression models for forecasting goals and match results in association football," International Journal of Forecasting, Elsevier, vol. 21(2), pages 331-340.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Gross, Johannes & Rebeggiani, Luca, 2018. "Chance or Ability? The Efficiency of the Football Betting Market Revisited," MPRA Paper 87230, University Library of Munich, Germany.
    2. Marc Garnica-Caparrós & Daniel Memmert & Fabian Wunderlich, 2022. "Artificial data in sports forecasting: a simulation framework for analysing predictive models in sports," Information Systems and e-Business Management, Springer, vol. 20(3), pages 551-580, September.
    3. Koopman, Siem Jan & Lit, Rutger, 2019. "Forecasting football match results in national league competitions using score-driven time series models," International Journal of Forecasting, Elsevier, vol. 35(2), pages 797-809.
    4. Hans Eetvelde & Lars Magnus Hvattum & Christophe Ley, 2023. "The Probabilistic Final Standing Calculator: a fair stochastic tool to handle abruptly stopped football seasons," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 107(1), pages 251-269, March.
    5. Chia-Hao Chang, 2021. "Construction of a Predictive Model for MLB Matches," Forecasting, MDPI, vol. 3(1), pages 1-11, February.
    6. Groll Andreas & Kneib Thomas & Mayr Andreas & Schauberger Gunther, 2018. "On the dependency of soccer scores – a sparse bivariate Poisson model for the UEFA European football championship 2016," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 14(2), pages 65-79, June.
    7. Babatunde Buraimo & David Peel & Rob Simmons, 2013. "Systematic Positive Expected Returns in the UK Fixed Odds Betting Market: An Analysis of the Fink Tank Predictions," IJFS, MDPI, vol. 1(4), pages 1-15, December.
    8. Baboota, Rahul & Kaur, Harleen, 2019. "Predictive analysis and modelling football results using machine learning approach for English Premier League," International Journal of Forecasting, Elsevier, vol. 35(2), pages 741-755.
    9. Lasek, Jan & Gagolewski, Marek, 2021. "Interpretable sports team rating models based on the gradient descent algorithm," International Journal of Forecasting, Elsevier, vol. 37(3), pages 1061-1071.
    10. Wheatcroft Edward, 2021. "Evaluating probabilistic forecasts of football matches: the case against the ranked probability score," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 17(4), pages 273-287, December.
    11. Lechner, Michael & Okasa, Gabriel, 2019. "Random Forest Estimation of the Ordered Choice Model," Economics Working Paper Series 1908, University of St. Gallen, School of Economics and Political Science.
    12. Zachary J. Smith & J. Eric Bickel, 2020. "Additive Scoring Rules for Discrete Sample Spaces," Decision Analysis, INFORMS, vol. 17(2), pages 115-133, June.
    13. Szczecinski Leszek, 2022. "G-Elo: generalization of the Elo algorithm by modeling the discretized margin of victory," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 18(1), pages 1-14, March.
    14. Wheatcroft, Edward, 2021. "Evaluating probabilistic forecasts of football matches: the case against the ranked probability score," LSE Research Online Documents on Economics 111494, London School of Economics and Political Science, LSE Library.
    15. Pearson Mitchell & Jr Glen Livingston & King Robert, 2020. "An exploration of predictive football modelling," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 16(1), pages 27-39, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stekler, H.O. & Sendor, David & Verlander, Richard, 2010. "Issues in sports forecasting," International Journal of Forecasting, Elsevier, vol. 26(3), pages 606-621, July.
      • Herman O. Stekler & David Sendor & Richard Verlander, 2009. "Issues in Sports Forecasting," Working Papers 2009-002, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    2. Singleton, Carl & Reade, J. James & Brown, Alasdair, 2020. "Going with your gut: The (In)accuracy of forecast revisions in a football score prediction game," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 89(C).
    3. O'Leary, Daniel E., 2017. "Crowd performance in prediction of the World Cup 2014," European Journal of Operational Research, Elsevier, vol. 260(2), pages 715-724.
    4. Brandt, Patrick T. & Freeman, John R. & Schrodt, Philip A., 2014. "Evaluating forecasts of political conflict dynamics," International Journal of Forecasting, Elsevier, vol. 30(4), pages 944-962.
    5. David Winkelmann & Marius Ötting & Christian Deutscher & Tomasz Makarewicz, 2024. "Are Betting Markets Inefficient? Evidence From Simulations and Real Data," Journal of Sports Economics, , vol. 25(1), pages 54-97, January.
    6. Wheatcroft Edward, 2021. "Evaluating probabilistic forecasts of football matches: the case against the ranked probability score," Journal of Quantitative Analysis in Sports, De Gruyter, vol. 17(4), pages 273-287, December.
    7. Wheatcroft, Edward, 2021. "Evaluating probabilistic forecasts of football matches: the case against the ranked probability score," LSE Research Online Documents on Economics 111494, London School of Economics and Political Science, LSE Library.
    8. Peter Dawson & Stephen Dobson & John Goddard & John Wilson, 2007. "Are football referees really biased and inconsistent?: evidence on the incidence of disciplinary sanction in the English Premier League," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(1), pages 231-250, January.
    9. Makridakis, Spyros & Hyndman, Rob J. & Petropoulos, Fotios, 2020. "Forecasting in social settings: The state of the art," International Journal of Forecasting, Elsevier, vol. 36(1), pages 15-28.
    10. Wunderlich, Fabian & Memmert, Daniel, 2020. "Are betting returns a useful measure of accuracy in (sports) forecasting?," International Journal of Forecasting, Elsevier, vol. 36(2), pages 713-722.
    11. Ian McHale & Rose Baker, 2014. "Econometric modelling of match results and scores," Chapters, in: John Goddard & Peter Sloane (ed.), Handbook on the Economics of Professional Football, chapter 9, pages 130-140, Edward Elgar Publishing.
    12. Martin Spann & Bernd Skiera, 2009. "Sports forecasting: a comparison of the forecast accuracy of prediction markets, betting odds and tipsters," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 28(1), pages 55-72.
    13. Adrian R. Bell & Chris Brooks & David Matthews & Charles Sutcliffe, 2012. "Over the moon or sick as a parrot? The effects of football results on a club's share price," Applied Economics, Taylor & Francis Journals, vol. 44(26), pages 3435-3452, September.
    14. Hvattum, Lars Magnus & Arntzen, Halvard, 2010. "Using ELO ratings for match result prediction in association football," International Journal of Forecasting, Elsevier, vol. 26(3), pages 460-470, July.
    15. J. James Reade & Carl Singleton & Alasdair Brown, 2021. "Evaluating strange forecasts: The curious case of football match scorelines," Scottish Journal of Political Economy, Scottish Economic Society, vol. 68(2), pages 261-285, May.
    16. Gross, Johannes & Rebeggiani, Luca, 2018. "Chance or Ability? The Efficiency of the Football Betting Market Revisited," MPRA Paper 87230, University Library of Munich, Germany.
    17. Bernardo, Giovanni & Ruberti, Massimo & Verona, Roberto, 2019. "Semi-strong inefficiency in the fixed odds betting market: Underestimating the positive impact of head coach replacement in the main European soccer leagues," The Quarterly Review of Economics and Finance, Elsevier, vol. 71(C), pages 239-246.
    18. J Reade & C Singleton & L Vaughan Williams, 2020. "Betting Markets for English Premier League Results and Scorelines: Evaluating a Simple Forecasting Model," Economic Issues Journal Articles, Economic Issues, vol. 25(1), pages 87-106, March.
    19. Mitchell, Karlyn & Pearce, Douglas K., 2007. "Professional forecasts of interest rates and exchange rates: Evidence from the Wall Street Journal's panel of economists," Journal of Macroeconomics, Elsevier, vol. 29(4), pages 840-854, December.
    20. Sung, Ming-Chien & McDonald, David C.J. & Johnson, Johnnie E.V. & Tai, Chung-Ching & Cheah, Eng-Tuck, 2019. "Improving prediction market forecasts by detecting and correcting possible over-reaction to price movements," European Journal of Operational Research, Elsevier, vol. 272(1), pages 389-405.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:jqsprt:v:8:y:2012:i:1:n:12. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyter.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.