Measuring forecasting accuracy: The case of judgmental adjustments to SKU-level demand forecasts
AbstractForecast adjustment commonly occurs when organizational forecasters adjust a statistical forecast of demand to take into account factors which are excluded from the statistical calculation. This paper addresses the question of how to measure the accuracy of such adjustments. We show that many existing error measures are generally not suited to the task, due to specific features of the demand data. Alongside the well-known weaknesses of existing measures, a number of additional effects are demonstrated that complicate the interpretation of measurement results and can even lead to false conclusions being drawn. In order to ensure an interpretable and unambiguous evaluation, we recommend the use of a metric based on aggregating performance ratios across time series using the weighted geometric mean. We illustrate that this measure has the advantage of treating over- and under-forecasting even-handedly, has a more symmetric distribution, and is robust.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
Bibliographic InfoArticle provided by Elsevier in its journal International Journal of Forecasting.
Volume (Year): 29 (2013)
Issue (Month): 3 ()
Contact details of provider:
Web page: http://www.elsevier.com/locate/ijforecast
Judgmental adjustments; Forecasting support systems; Forecast accuracy; Forecast evaluation; Forecast error measures;
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Fildes, Robert & Goodwin, Paul & Lawrence, Michael & Nikolopoulos, Konstantinos, 2009. "Effective forecasting and judgmental adjustments: an empirical evaluation and strategies for improvement in supply-chain planning," International Journal of Forecasting, Elsevier, vol. 25(1), pages 3-23.
- Fildes, Robert, 1992. "The evaluation of extrapolative forecasting methods," International Journal of Forecasting, Elsevier, vol. 8(1), pages 81-98, June.
- Rob J. Hyndman, 2006. "Another Look at Forecast Accuracy Metrics for Intermittent Demand," Foresight: The International Journal of Applied Forecasting, International Institute of Forecasters, issue 4, pages 43-46, June.
- Syntetos, Aris A. & Boylan, John E., 2005. "The accuracy of intermittent demand estimates," International Journal of Forecasting, Elsevier, vol. 21(2), pages 303-314.
- Robalo Marques, Carlos & Duarte Neves, Pedro & Morais Sarmento, Luis, 2003.
"Evaluating core inflation indicators,"
Elsevier, vol. 20(4), pages 765-775, July.
- Pedro Duarte Neves & Luís Morais Sarmento & Carlos Robalo Marques, 1999. "Evaluating core inflation indicators," Economic Bulletin and Financial Stability Report Articles, Banco de Portugal, Economics and Research Department.
- Jim Hoover, 2006. "Measuring Forecast Accuracy: Omissions in Today's Forecasting Engines and Demand-Planning Software," Foresight: The International Journal of Applied Forecasting, International Institute of Forecasters, issue 4, pages 32-35, June.
- Stephan Kolassa & Wolfgang Schütz, 2007. "Advantages of the MAD/Mean Ratio over the MAPE," Foresight: The International Journal of Applied Forecasting, International Institute of Forecasters, issue 6, pages 40-43, Spring.
- Rob J. Hyndman & Yeasmin Khandakar, .
"Automatic Time Series Forecasting: The forecast Package for R,"
Journal of Statistical Software,
American Statistical Association, vol. 27(i03).
- Rob J. Hyndman & Yeasmin Khandakar, 2007. "Automatic time series forecasting: the forecast package for R," Monash Econometrics and Business Statistics Working Papers 6/07, Monash University, Department of Econometrics and Business Statistics.
- Philip Hans Franses & Rianne Legerstee, 2010. "Do experts' adjustments on model-based SKU-level forecasts improve forecast quality?," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 29(3), pages 331-340.
- Goodwin, Paul & Lawton, Richard, 1999. "On the asymmetry of the symmetric MAPE," International Journal of Forecasting, Elsevier, vol. 15(4), pages 405-408, October.
- Makridakis, Spyros, 1993. "Accuracy measures: theoretical and practical concerns," International Journal of Forecasting, Elsevier, vol. 9(4), pages 527-529, December.
- Rob J. Hyndman & Anne B. Koehler, 2005.
"Another Look at Measures of Forecast Accuracy,"
Monash Econometrics and Business Statistics Working Papers
13/05, Monash University, Department of Econometrics and Business Statistics.
- Donna F. Davis & John T. Mentzer & Teresa M. Mccarthy & Susan L. Golicic, 2006. "The evolution of sales forecasting management: a 20-year longitudinal study of forecasting practices," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 25(5), pages 303-324.
- Armstrong, J. Scott & Collopy, Fred, 1992. "Error measures for generalizing about forecasting methods: Empirical comparisons," International Journal of Forecasting, Elsevier, vol. 8(1), pages 69-80, June.
- Zellner, Arnold, 1986. "A tale of forecasting 1001 series : The Bayesian knight strikes again," International Journal of Forecasting, Elsevier, vol. 2(4), pages 491-494.
- Sbrana, Giacomo & Silvestrini, Andrea, 2013.
"Forecasting aggregate demand: Analytical comparison of top-down and bottom-up approaches in a multivariate exponential smoothing framework,"
International Journal of Production Economics,
Elsevier, vol. 146(1), pages 185-198.
- Giacomo Sbrana & Andrea Silvestrini, 2013. "Forecasting aggregate demand: analytical comparison of top-down and bottom-up approaches in a multivariate exponential smoothing framework," Temi di discussione (Economic working papers) 929, Bank of Italy, Economic Research and International Relations Area.
- Fildes, Robert & Petropoulos, Fotios, 2013. "An evaluation of simple forecasting model selection rules," MPRA Paper 51772, University Library of Munich, Germany.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Zhang, Lei).
If references are entirely missing, you can add them using this form.