IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2411.17743.html
   My bibliography  Save this paper

Ranking probabilistic forecasting models with different loss functions

Author

Listed:
  • Tomasz Serafin
  • Bartosz Uniejewski

Abstract

In this study, we introduced various statistical performance metrics, based on the pinball loss and the empirical coverage, for the ranking of probabilistic forecasting models. We tested the ability of the proposed metrics to determine the top performing forecasting model and investigated the use of which metric corresponds to the highest average per-trade profit in the out-of-sample period. Our findings show that for the considered trading strategy, ranking the forecasting models according to the coverage of quantile forecasts used in the trading hours exhibits a superior economic performance.

Suggested Citation

  • Tomasz Serafin & Bartosz Uniejewski, 2024. "Ranking probabilistic forecasting models with different loss functions," Papers 2411.17743, arXiv.org.
    • Handle: RePEc:arx:papers:2411.17743
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2411.17743
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Marcelo Fernandes & Emmanuel Guerre & Eduardo Horta, 2021. "Smoothing Quantile Regressions," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 338-357, January.
    2. Marcjasz, Grzegorz & Narajewski, Michał & Weron, Rafał & Ziel, Florian, 2023. "Distributional neural networks for electricity price forecasting," Energy Economics, Elsevier, vol. 125(C).
    3. Kath, Christopher & Ziel, Florian, 2021. "Conformal prediction interval estimation and applications to day-ahead and intraday power markets," International Journal of Forecasting, Elsevier, vol. 37(2), pages 777-799.
    4. Jakub Nowotarski & Rafał Weron, 2015. "Computing electricity spot price prediction intervals using quantile regression and forecast averaging," Computational Statistics, Springer, vol. 30(3), pages 791-803, September.
    5. Christopher Kath & Florian Ziel, 2018. "The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts," Papers 1811.08604, arXiv.org.
    6. Bartosz Uniejewski, 2023. "Smoothing Quantile Regression Averaging: A new approach to probabilistic forecasting of electricity prices," Papers 2302.00411, arXiv.org, revised Nov 2024.
    7. Chatfield, Chris, 1993. "Calculating Interval Forecasts: Reply," Journal of Business & Economic Statistics, American Statistical Association, vol. 11(2), pages 143-144, April.
    8. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    9. Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose & Robert L. Winkler, 2017. "Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs," Operations Research, INFORMS, vol. 65(3), pages 712-728, June.
    10. Chatfield, Chris, 1993. "Calculating Interval Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 11(2), pages 121-135, April.
    11. Marcjasz, Grzegorz & Uniejewski, Bartosz & Weron, Rafał, 2020. "Probabilistic electricity price forecasting with NARX networks: Combine point or probabilistic forecasts?," International Journal of Forecasting, Elsevier, vol. 36(2), pages 466-479.
    12. Kath, Christopher & Ziel, Florian, 2018. "The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts," Energy Economics, Elsevier, vol. 76(C), pages 411-423.
    13. Uniejewski, Bartosz & Weron, Rafał, 2021. "Regularized quantile regression averaging for probabilistic electricity price forecasting," Energy Economics, Elsevier, vol. 95(C).
    14. Katarzyna Maciejowska & Tomasz Serafin & Bartosz Uniejewski, 2023. "Probabilistic forecasting with a hybrid Factor-QRA approach: Application to electricity trading," Papers 2303.08565, arXiv.org, revised Nov 2024.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Katarzyna Maciejowska & Bartosz Uniejewski & Rafa{l} Weron, 2022. "Forecasting Electricity Prices," Papers 2204.11735, arXiv.org.
    2. Bartosz Uniejewski, 2023. "Smoothing Quantile Regression Averaging: A new approach to probabilistic forecasting of electricity prices," Papers 2302.00411, arXiv.org, revised Nov 2024.
    3. Lipiecki, Arkadiusz & Uniejewski, Bartosz & Weron, Rafał, 2024. "Postprocessing of point predictions for probabilistic forecasting of day-ahead electricity prices: The benefits of using isotonic distributional regression," Energy Economics, Elsevier, vol. 139(C).
    4. Agakishiev, Ilyas & Härdle, Wolfgang Karl & Kopa, Milos & Kozmik, Karel & Petukhina, Alla, 2025. "Multivariate probabilistic forecasting of electricity prices with trading applications," Energy Economics, Elsevier, vol. 141(C).
    5. Weronika Nitka & Rafał Weron, 2023. "Combining predictive distributions of electricity prices. Does minimizing the CRPS lead to optimal decisions in day-ahead bidding?," Operations Research and Decisions, Wroclaw University of Science and Technology, Faculty of Management, vol. 33(3), pages 105-118.
    6. Nickelsen, Daniel & Müller, Gernot, 2025. "Bayesian hierarchical probabilistic forecasting of intraday electricity prices," Applied Energy, Elsevier, vol. 380(C).
    7. Serafin, Tomasz & Marcjasz, Grzegorz & Weron, Rafał, 2022. "Trading on short-term path forecasts of intraday electricity prices," Energy Economics, Elsevier, vol. 112(C).
    8. Stephen Haben & Julien Caudron & Jake Verma, 2021. "Probabilistic Day-Ahead Wholesale Price Forecast: A Case Study in Great Britain," Forecasting, MDPI, vol. 3(3), pages 1-37, August.
    9. Tomasz Serafin & Bartosz Uniejewski & Rafał Weron, 2019. "Averaging Predictive Distributions Across Calibration Windows for Day-Ahead Electricity Price Forecasting," Energies, MDPI, vol. 12(13), pages 1-12, July.
    10. Micha{l} Narajewski & Florian Ziel, 2020. "Ensemble Forecasting for Intraday Electricity Prices: Simulating Trajectories," Papers 2005.01365, arXiv.org, revised Aug 2020.
    11. Hilger, Hannes & Witthaut, Dirk & Dahmen, Manuel & Rydin Gorjão, Leonardo & Trebbien, Julius & Cramer, Eike, 2024. "Multivariate scenario generation of day-ahead electricity prices using normalizing flows," Applied Energy, Elsevier, vol. 367(C).
    12. Marcjasz, Grzegorz & Narajewski, Michał & Weron, Rafał & Ziel, Florian, 2023. "Distributional neural networks for electricity price forecasting," Energy Economics, Elsevier, vol. 125(C).
    13. Jiang, Ping & Nie, Ying & Wang, Jianzhou & Huang, Xiaojia, 2023. "Multivariable short-term electricity price forecasting using artificial intelligence and multi-input multi-output scheme," Energy Economics, Elsevier, vol. 117(C).
    14. Cornell, Cameron & Dinh, Nam Trong & Pourmousavi, S. Ali, 2024. "A probabilistic forecast methodology for volatile electricity prices in the Australian National Electricity Market," International Journal of Forecasting, Elsevier, vol. 40(4), pages 1421-1437.
    15. Michał Narajewski, 2022. "Probabilistic Forecasting of German Electricity Imbalance Prices," Energies, MDPI, vol. 15(14), pages 1-17, July.
    16. Katarzyna Maciejowska & Weronika Nitka, 2024. "Multiple split approach -- multidimensional probabilistic forecasting of electricity markets," Papers 2407.07795, arXiv.org.
    17. Jozef Barunik & Lubos Hanus, 2023. "Learning Probability Distributions of Day-Ahead Electricity Prices," Papers 2310.02867, arXiv.org, revised Oct 2023.
    18. Katarzyna Chk{e}'c & Bartosz Uniejewski & Rafa{l} Weron, 2025. "Extrapolating the long-term seasonal component of electricity prices for forecasting in the day-ahead market," Papers 2503.02518, arXiv.org.
    19. Uniejewski, Bartosz & Maciejowska, Katarzyna, 2023. "LASSO principal component averaging: A fully automated approach for point forecast pooling," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1839-1852.
    20. Simon Hirsch, 2025. "Online Multivariate Regularized Distributional Regression for High-dimensional Probabilistic Electricity Price Forecasting," Papers 2504.02518, arXiv.org.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2411.17743. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.