IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/11-03.html
   My bibliography  Save this paper

Using Model Selection Algorthims to Obtain Reliable Coefficient Estimates

Author

Listed:

Abstract

This review surveys a number of common Model Selection Algorithms (MSAs), discusses how they relate to each other, and identifies factors that explain their relative performances. At the heart of MSA performance is the trade-off between Type I and Type II errors. Some relevant variables will be mistakenly excluded, and some irrelevant variables will be retained by chance. A successful MSA will find the optimal trade-off between the two types of errors for a given data environment. Whether a given MSA will be successful in a given environment depends on the relative costs of these two types of errors. We use Monte Carlo experimentation to illustrate these issues. We confirm that no MSA does best in all circumstances. Even the worst MSA in terms of overall performance – the strategy of including all candidate variables – sometimes performs best (viz., when all candidate variables are relevant). We also show how (i) the ratio of relevant to total candidate variables and (ii) DGP noise affect relative MSA performance. Finally, we discuss a number of issues complicating the task of MSAs in producing reliable coefficient estimates.

Suggested Citation

  • Jennifer Castle & Xiaochuan Qin & W. Robert Reed, 2011. "Using Model Selection Algorthims to Obtain Reliable Coefficient Estimates," Working Papers in Economics 11/03, University of Canterbury, Department of Economics and Finance.
    • Handle: RePEc:cbt:econwp:11/03
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/1103.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Castle Jennifer L. & Doornik Jurgen A & Hendry David F., 2011. "Evaluating Automatic Model Selection," Journal of Time Series Econometrics, De Gruyter, vol. 3(1), pages 1-33, February.
    2. Kevin D. Hoover & Stephen J. Perez, 1999. "Data mining reconsidered: encompassing and the general-to-specific approach to specification search," Econometrics Journal, Royal Economic Society, vol. 2(2), pages 167-191.
    3. Phillips, Peter C.B., 2005. "Automated Discovery In Econometrics," Econometric Theory, Cambridge University Press, vol. 21(1), pages 3-20, February.
    4. Gourieroux,Christian & Monfort,Alain, 1995. "Statistics and Econometric Models," Cambridge Books, Cambridge University Press, number 9780521471626.
    5. Peter C.B. Phillips, 1995. "Automated Forecasts of Asia-Pacific Economic Activity," Cowles Foundation Discussion Papers 1103, Cowles Foundation for Research in Economics, Yale University.
    6. McQuarrie, Allan D., 1999. "A small-sample correction for the Schwarz SIC model selection criterion," Statistics & Probability Letters, Elsevier, vol. 44(1), pages 79-86, August.
    7. David F. Hendry & Hans-Martin Krolzig, 2005. "The Properties of Automatic "GETS" Modelling," Economic Journal, Royal Economic Society, vol. 115(502), pages 32-61, March.
    8. Pagan, Adrian, 1987. "Three Econometric Methodologies: A Critical Appraisal," Journal of Economic Surveys, Wiley Blackwell, vol. 1(1), pages 3-24.
    9. Xavier Sala-I-Martin & Gernot Doppelhofer & Ronald I. Miller, 2004. "Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach," American Economic Review, American Economic Association, vol. 94(4), pages 813-835, September.
    10. Julia Campos & David F. Hendry & Hans‐Martin Krolzig, 2003. "Consistent Model Selection by an Automatic Gets Approach," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 65(s1), pages 803-819, December.
    11. Beck, Nathaniel & Katz, Jonathan N., 1995. "What To Do (and Not to Do) with Time-Series Cross-Section Data," American Political Science Review, Cambridge University Press, vol. 89(3), pages 634-647, September.
    12. David F. Hendry & Hans‐Martin Krolzig, 2004. "We Ran One Regression," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 66(5), pages 799-810, December.
    13. David F. Hendry & Hans-Martin Krolzig, 2004. "We Ran One Regression," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 66(5), pages 799-810, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Durevall, Dick & Loening, Josef L. & Ayalew Birru, Yohannes, 2013. "Inflation dynamics and food prices in Ethiopia," Journal of Development Economics, Elsevier, vol. 104(C), pages 89-106.
    2. W. Robert Reed, 2018. "A Primer on the ‘Reproducibility Crisis’ and Ways to Fix It," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 51(2), pages 286-300, June.
    3. Ragnar Nymoen & Kari Pedersen & Jon Ivar Sjåberg, 2019. "Estimation of Effects of Recent Macroprudential Policies in a Sample of Advanced Open Economies," IJFS, MDPI, vol. 7(2), pages 1-20, May.
    4. Hendry, David F., 2018. "Deciding between alternative approaches in macroeconomics," International Journal of Forecasting, Elsevier, vol. 34(1), pages 119-135.
    5. Nymoen, Ragnar & Pedersen, Kari & Sjåberg, Jon Ivar, 2018. "Estimation of effects of recent macroprudential policies in a sample of advanced open economies," Memorandum 5/2018, Oslo University, Department of Economics.
    6. Cunha, Ronan & Pereira, Pedro L. Valls, 2015. "Automatic model selection for forecasting Brazilian stock returns," Textos para discussão 398, FGV EESP - Escola de Economia de São Paulo, Fundação Getulio Vargas (Brazil).
    7. Kevin S. Nell, 2018. "Conditional Divergence in the Post-1989 Globalisation Period," CEF.UP Working Papers 1806, Universidade do Porto, Faculdade de Economia do Porto.
    8. Ericsson Neil R., 2016. "Testing for and estimating structural breaks and other nonlinearities in a dynamic monetary sector," Studies in Nonlinear Dynamics & Econometrics, De Gruyter, vol. 20(4), pages 377-398, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Castle Jennifer L. & Doornik Jurgen A & Hendry David F., 2011. "Evaluating Automatic Model Selection," Journal of Time Series Econometrics, De Gruyter, vol. 3(1), pages 1-33, February.
    2. Julia Campos & Neil R. Ericsson & David F. Hendry, 2005. "General-to-specific modeling: an overview and selected bibliography," International Finance Discussion Papers 838, Board of Governors of the Federal Reserve System (U.S.).
    3. Mark F. J. Steel, 2020. "Model Averaging and Its Use in Economics," Journal of Economic Literature, American Economic Association, vol. 58(3), pages 644-719, September.
    4. Castle, Jennifer L. & Doornik, Jurgen A. & Hendry, David F., 2012. "Model selection when there are multiple breaks," Journal of Econometrics, Elsevier, vol. 169(2), pages 239-246.
    5. R Burger & S du Plessis, 2011. "Examining the Robustness of Competing Explanations of Slow Growth in African Countries," Studies in Economics and Econometrics, Taylor & Francis Journals, vol. 35(3), pages 21-47, December.
    6. Camila Epprecht & Dominique Guegan & Álvaro Veiga & Joel Correa da Rosa, 2017. "Variable selection and forecasting via automated methods for linear models: LASSO/adaLASSO and Autometrics," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-00917797, HAL.
    7. W. Robert Reed, 2009. "The Determinants Of U.S. State Economic Growth: A Less Extreme Bounds Analysis," Economic Inquiry, Western Economic Association International, vol. 47(4), pages 685-700, October.
    8. Ulaşan, Bülent, 2011. "Cross-country growth empirics and model uncertainty: An overview," Economics Discussion Papers 2011-37, Kiel Institute for the World Economy (IfW Kiel).
    9. Camila Epprecht & Dominique Guegan & Álvaro Veiga, 2013. "Comparing variable selection techniques for linear regression: LASSO and Autometrics," Documents de travail du Centre d'Economie de la Sorbonne 13080, Université Panthéon-Sorbonne (Paris 1), Centre d'Economie de la Sorbonne.
    10. David F. Hendry & Hans‐Martin Krolzig, 2004. "We Ran One Regression," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 66(5), pages 799-810, December.
    11. Melisa Chanegriha & Chris Stewart & Christopher Tsoukis, 2017. "Identifying the robust economic, geographical and political determinants of FDI: an Extreme Bounds Analysis," Empirical Economics, Springer, vol. 52(2), pages 759-776, March.
    12. Antonio Ciccone & Marek Jarociński, 2010. "Determinants of Economic Growth: Will Data Tell?," American Economic Journal: Macroeconomics, American Economic Association, vol. 2(4), pages 222-246, October.
    13. Sai Ding & John Knight, 2011. "Why has China Grown So Fast? The Role of Physical and Human Capital Formation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 73(2), pages 141-174, April.
    14. David F. Hendry & Grayham E. Mizon, 2016. "Improving the teaching of econometrics," Cogent Economics & Finance, Taylor & Francis Journals, vol. 4(1), pages 1170096-117, December.
    15. Yongfu Huang, 2005. "What determines financial development?," Bristol Economics Discussion Papers 05/580, School of Economics, University of Bristol, UK.
    16. Jennifer Castle & David Hendry, 2010. "Automatic Selection for Non-linear Models," Economics Series Working Papers 473, University of Oxford, Department of Economics.
    17. John Knight & Sai Ding, 2008. "Why has China Grown so Fast? The Role of Structural Change," Economics Series Working Papers 415, University of Oxford, Department of Economics.
    18. Schneider Ulrike & Wagner Martin, 2012. "Catching Growth Determinants with the Adaptive Lasso," German Economic Review, De Gruyter, vol. 13(1), pages 71-85, February.
    19. Neil R. Ericsson, 2008. "The Fragility of Sensitivity Analysis: An Encompassing Perspective," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(s1), pages 895-914, December.
    20. Jennifer L. Castle & Jurgen A. Doornik & David F. Hendry, 2013. "Model Selection in Equations with Many ‘Small’ Effects," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 75(1), pages 6-22, February.

    More about this item

    Keywords

    Model selection algorithms; Information Criteria; General-to-Specific modeling; Bayesian Model Averaging; Portfolio Models; AIC; SIC; AICc; SICc; Monte Carlo Analysis; Autometrics;
    All these keywords.

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:11/03. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.