IDEAS home Printed from https://ideas.repec.org/a/spr/testjl/v30y2021i3d10.1007_s11749-020-00737-9.html
   My bibliography  Save this article

Simple measures of uncertainty for model selection

Author

Listed:
  • Xiaohui Liu

    (Jiangxi University of Finance and Economics)

  • Yuanyuan Li

    (University of California, Davis)

  • Jiming Jiang

    (University of California, Davis)

Abstract

We develop two simple measures of uncertainty for a model selection procedure. The first measure is similar in spirit to confidence set in parameter estimation; the second measure is focusing on error in model selection. The proposed methods are simpler, both conceptually and computationally, than the existing measures of uncertainty in model selection. We recognize major differences between model selection and traditional estimation or prediction problems, and propose reasonable frameworks, under which these measures are developed, and their theoretical properties are established. Empirical studies demonstrate performance of the proposed measures, their superiority over the existing methods, and their relevance to real-life applications.

Suggested Citation

  • Xiaohui Liu & Yuanyuan Li & Jiming Jiang, 2021. "Simple measures of uncertainty for model selection," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 30(3), pages 673-692, September.
    • Handle: RePEc:spr:testjl:v:30:y:2021:i:3:d:10.1007_s11749-020-00737-9
    DOI: 10.1007/s11749-020-00737-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11749-020-00737-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11749-020-00737-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yuan, Zheng & Yang, Yuhong, 2005. "Combining Linear Regression Models: When and How?," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 1202-1214, December.
    2. Peter R. Hansen & Asger Lunde & James M. Nason, 2011. "The Model Confidence Set," Econometrica, Econometric Society, vol. 79(2), pages 453-497, March.
    3. Chao Zheng & Davide Ferrari & Michael Zhang & Paul Baird, 2019. "Ranking the importance of genetic factors by variable‐selection confidence sets," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 68(3), pages 727-749, April.
    4. Datta, Gauri S. & Hall, Peter & Mandal, Abhyuday, 2011. "Model Selection by Testing for the Presence of Small-Area Effects, and Application to Area-Level Data," Journal of the American Statistical Association, American Statistical Association, vol. 106(493), pages 362-374.
    5. Claeskens,Gerda & Hjort,Nils Lid, 2008. "Model Selection and Model Averaging," Cambridge Books, Cambridge University Press, number 9780521852258.
    6. Zou, Hui, 2006. "The Adaptive Lasso and Its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1418-1429, December.
    7. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    8. Xie, Minge & Singh, Kesar & Zhang, Cun-Hui, 2009. "Confidence Intervals for Population Ranks in the Presence of Ties and Near Ties," Journal of the American Statistical Association, American Statistical Association, vol. 104(486), pages 775-788.
    9. Xiaotong Shen & Wei Pan & Yunzhang Zhu, 2012. "Likelihood-Based Selection and Sharp Parameter Estimation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(497), pages 223-232, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Faguang Wen & Jiming Jiang & Yihui Luan, 2024. "Model Selection Path and Construction of Model Confidence Set under High-Dimensional Variables," Mathematics, MDPI, vol. 12(5), pages 1-21, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Liu, Chu-An, 2015. "Distribution theory of the least squares averaging estimator," Journal of Econometrics, Elsevier, vol. 186(1), pages 142-159.
    2. Philippe Goulet Coulombe & Maxime Leroux & Dalibor Stevanovic & Stéphane Surprenant, 2022. "How is machine learning useful for macroeconomic forecasting?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 920-964, August.
    3. Mark F. J. Steel, 2020. "Model Averaging and Its Use in Economics," Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.
    4. Niko Hauzenberger & Michael Pfarrhofer & Luca Rossini, 2020. "Sparse time-varying parameter VECMs with an application to modeling electricity prices," Papers 2011.04577, arXiv.org, revised Apr 2023.
    5. Liu, Chu-An, 2012. "A plug-in averaging estimator for regressions with heteroskedastic errors," MPRA Paper 41414, University Library of Munich, Germany.
    6. Yuying Sun & Shaoxin Hong & Zongwu Cai, 2023. "Optimal Local Model Averaging for Divergent-Dimensional Functional-Coefficient Regressions," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202309, University of Kansas, Department of Economics, revised Sep 2023.
    7. Faguang Wen & Jiming Jiang & Yihui Luan, 2024. "Model Selection Path and Construction of Model Confidence Set under High-Dimensional Variables," Mathematics, MDPI, vol. 12(5), pages 1-21, February.
    8. Aman Ullah & Huansha Wang, 2013. "Parametric and Nonparametric Frequentist Model Selection and Model Averaging," Econometrics, MDPI, vol. 1(2), pages 1-23, September.
    9. Qin, Yichen & Wang, Linna & Li, Yang & Li, Rong, 2023. "Visualization and assessment of model selection uncertainty," Computational Statistics & Data Analysis, Elsevier, vol. 178(C).
    10. Yize Zhao & Matthias Chung & Brent A. Johnson & Carlos S. Moreno & Qi Long, 2016. "Hierarchical Feature Selection Incorporating Known and Novel Biological Information: Identifying Genomic Features Related to Prostate Cancer Recurrence," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1427-1439, October.
    11. Andrea Bucci, 2020. "Realized Volatility Forecasting with Neural Networks," Journal of Financial Econometrics, Oxford University Press, vol. 18(3), pages 502-531.
    12. Angelica Gianfreda & Francesco Ravazzolo & Luca Rossini, 2020. "Large Time-Varying Volatility Models for Electricity Prices," Working Papers No 05/2020, Centre for Applied Macro- and Petroleum economics (CAMP), BI Norwegian Business School.
    13. Naimoli, Antonio, 2022. "Modelling the persistence of Covid-19 positivity rate in Italy," Socio-Economic Planning Sciences, Elsevier, vol. 82(PA).
    14. Xiaotong Shen & Wei Pan & Yunzhang Zhu & Hui Zhou, 2013. "On constrained and regularized high-dimensional regression," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 65(5), pages 807-832, October.
    15. Costa, Alexandre Bonnet R. & Ferreira, Pedro Cavalcanti G. & Gaglianone, Wagner P. & Guillén, Osmani Teixeira C. & Issler, João Victor & Lin, Yihao, 2021. "Machine learning and oil price point and density forecasting," Energy Economics, Elsevier, vol. 102(C).
    16. Tomáš Plíhal, 2021. "Scheduled macroeconomic news announcements and Forex volatility forecasting," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(8), pages 1379-1397, December.
    17. Nowotarski, Jakub & Weron, Rafał, 2018. "Recent advances in electricity price forecasting: A review of probabilistic forecasting," Renewable and Sustainable Energy Reviews, Elsevier, vol. 81(P1), pages 1548-1568.
    18. Hiraki, Kazuhiro & Sun, Chuanping, 2022. "A toolkit for exploiting contemporaneous stock correlations," Journal of Empirical Finance, Elsevier, vol. 65(C), pages 99-124.
    19. Simona Buscemi & Antonella Plaia, 2020. "Model selection in linear mixed-effect models," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 104(4), pages 529-575, December.
    20. Canhong Wen & Xueqin Wang & Shaoli Wang, 2015. "Laplace Error Penalty-based Variable Selection in High Dimension," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 42(3), pages 685-700, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:testjl:v:30:y:2021:i:3:d:10.1007_s11749-020-00737-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.