IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/09-13.html
   My bibliography  Save this paper

How To Pick The Best Regression Equation: A Review And Comparison Of Model Selection Algorithms

Author

Listed:

Abstract

This paper reviews and compares twenty-one different model selection algorithms (MSAs) representing a diversity of approaches, including (i) information criteria such as AIC and SIC; (ii) selection of a “portfolio” or best subset of models; (iii) general-to-specific algorithms, (iv) forward-stepwise regression approaches; (v) Bayesian Model Averaging; and (vi) inclusion of all variables. We use coefficient unconditional mean-squared error (UMSE) as the basis for our measure of MSA performance. Our main goal is to identify the factors that determine MSA performance. Towards this end, we conduct Monte Carlo experiments across a variety of data environments. Our experiments show that MSAs differ substantially with respect to their performance on relevant and irrelevant variables. We relate this to their associated penalty functions, and a bias-variance tradeoff in coefficient estimates. It follows that no MSA will dominate under all conditions. However, when we restrict our analysis to conditions where automatic variable selection is likely to be of greatest value, we find that two general-to-specific MSAs, Autometrics, do as well or better than all others in over 90% of the experiments.

Suggested Citation

  • Jennifer L. Castle & Xiaochuan Qin & W. Robert Reed, 2009. "How To Pick The Best Regression Equation: A Review And Comparison Of Model Selection Algorithms," Working Papers in Economics 09/13, University of Canterbury, Department of Economics and Finance.
  • Handle: RePEc:cbt:econwp:09/13
    as

    Download full text from publisher

    File URL: http://www.econ.canterbury.ac.nz/RePEc/cbt/econwp/0913.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Ernst Fehr & Simon Gächter, 2000. "Fairness and Retaliation: The Economics of Reciprocity," Journal of Economic Perspectives, American Economic Association, vol. 14(3), pages 159-181, Summer.
    2. Nikos Nikiforakis & Dirk Engelmann, 2008. "Feuds in the Laboratory? A Social Dilemma Experiment," Department of Economics - Working Papers Series 1058, The University of Melbourne.
    3. Charness, Gary & haruvy, Ernan & Sonsino, Doron, 2001. "Social Distance and Reciprocity: The Internet vs. the Laboratory," University of California at Santa Barbara, Economics Working Paper Series qt3dt073wb, Department of Economics, UC Santa Barbara.
    4. Catherine C. Eckel & Philip J. Grossman, 2008. "Forecasting Risk Attitudes: An Experimental Study Using Actual and Forecast Gamble Choices," Monash Economics Working Papers archive-01, Monash University, Department of Economics.
    5. Gary Charness & Matthew Rabin, 2002. "Understanding Social Preferences with Simple Tests," The Quarterly Journal of Economics, Oxford University Press, pages 817-869.
    6. Ernst Fehr & Klaus M. Schmidt, 1999. "A Theory of Fairness, Competition, and Cooperation," The Quarterly Journal of Economics, Oxford University Press, vol. 114(3), pages 817-868.
    7. Dürsch, Peter & Servátka, Maros, 2007. "Risky punishment and reward in the prisoner's dilemma," Papers 07-62, Sonderforschungsbreich 504.
    8. Dufwenberg, Martin & Kirchsteiger, Georg, 2004. "A theory of sequential reciprocity," Games and Economic Behavior, Elsevier, vol. 47(2), pages 268-298, May.
    9. Simon Gachter & Ernst Fehr, 2000. "Cooperation and Punishment in Public Goods Experiments," American Economic Review, American Economic Association, pages 980-994.
    10. David Masclet & Charles Noussair & Steven Tucker & Marie-Claire Villeval, 2003. "Monetary and Nonmonetary Punishment in the Voluntary Contributions Mechanism," American Economic Review, American Economic Association, pages 366-380.
    11. James C. Cox & Daniel Friedman & Vjollca Sadiraj, 2008. "Revealed Altruism," Econometrica, Econometric Society, vol. 76(1), pages 31-69, January.
    12. Gangadharan, Lata & Nikiforakis, Nikos, 2009. "Does the size of the action set matter for cooperation?," Economics Letters, Elsevier, pages 115-117.
    13. Robert H. Frank & Thomas Gilovich & Dennis T. Regan, 1993. "Does Studying Economics Inhibit Cooperation?," Journal of Economic Perspectives, American Economic Association, pages 159-171.
    14. Falk, Armin & Fischbacher, Urs, 2006. "A theory of reciprocity," Games and Economic Behavior, Elsevier, pages 293-315.
    15. Gary S. Becker, 1974. "Crime and Punishment: An Economic Approach," NBER Chapters,in: Essays in the Economics of Crime and Punishment, pages 1-54 National Bureau of Economic Research, Inc.
    16. Nikos Nikiforakis & Hans-Theo Normann, 2008. "A comparative statics analysis of punishment in public-good experiments," Experimental Economics, Springer;Economic Science Association, vol. 11(4), pages 358-369, December.
    17. Ernst Fehr & John A. List, 2004. "The Hidden Costs and Returns of Incentives-Trust and Trustworthiness Among CEOs," Journal of the European Economic Association, MIT Press, pages 743-771.
    18. Charles A. Holt & Susan K. Laury, 2002. "Risk Aversion and Incentive Effects," American Economic Review, American Economic Association, pages 1644-1655.
    19. Cox, James C & Smith, Vernon L & Walker, James M, 1988. "Theory and Individual Behavior of First-Price Auctions," Journal of Risk and Uncertainty, Springer, vol. 1(1), pages 61-99, March.
    20. Nikiforakis, Nikos, 2008. "Punishment and counter-punishment in public good games: Can we really govern ourselves," Journal of Public Economics, Elsevier, pages 91-112.
    21. Michael Kosfeld & Armin Falk, 2006. "The Hidden Costs of Control," American Economic Review, American Economic Association, vol. 96(5), pages 1611-1630, December.
    22. Laurent Denant-Boemont & David Masclet & Charles Noussair, 2007. "Punishment, counterpunishment and sanction enforcement in a social dilemma experiment," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), pages 145-167.
    23. Carpenter, Jeffrey P., 2007. "The demand for punishment," Journal of Economic Behavior & Organization, Elsevier, pages 522-542.
    24. Cox, James C. & Friedman, Daniel & Gjerstad, Steven, 2007. "A tractable model of reciprocity and fairness," Games and Economic Behavior, Elsevier, vol. 59(1), pages 17-45, April.
    25. Anderson, Christopher M. & Putterman, Louis, 2006. "Do non-strategic sanctions obey the law of demand? The demand for punishment in the voluntary contribution mechanism," Games and Economic Behavior, Elsevier, pages 1-24.
    26. Harrison, Glenn W., 1986. "An experimental test for risk aversion," Economics Letters, Elsevier, vol. 21(1), pages 7-11.
    27. Charles Noussair & Steven Tucker, 2005. "Combining Monetary and Social Sanctions to Promote Cooperation," Economic Inquiry, Western Economic Association International, vol. 43(3), pages 649-660, July.
    28. Marwell, Gerald & Ames, Ruth E., 1981. "Economists free ride, does anyone else? : Experiments on the provision of public goods, IV," Journal of Public Economics, Elsevier, pages 295-310.
    29. John R. Carter & Michael D. Irons, 1991. "Are Economists Different, and If So, Why?," Journal of Economic Perspectives, American Economic Association, pages 171-177.
    30. Roth, Alvin E, 1988. "Laboratory Experimentation in Economics: A Methodological Overview," Economic Journal, Royal Economic Society, vol. 98(393), pages 974-1031, December.
    31. Frey, Bruno S & Jegen, Reto, 2001. " Motivation Crowding Theory," Journal of Economic Surveys, Wiley Blackwell, vol. 15(5), pages 589-611, December.
    32. Isaac, R Mark & James, Duncan, 2000. "Just Who Are You Calling Risk Averse?," Journal of Risk and Uncertainty, Springer, vol. 20(2), pages 177-187, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Alex Mandilaras & Graham Bird, 2008. "Foreign exchange pressures in Latin America: does debt matter?," Journal of International Development, John Wiley & Sons, Ltd., pages 613-627.
    2. Castle Jennifer L. & Doornik Jurgen A & Hendry David F., 2011. "Evaluating Automatic Model Selection," Journal of Time Series Econometrics, De Gruyter, pages 1-33.
    3. Steven L. Scott & Hal R. Varian, 2015. "Bayesian Variable Selection for Nowcasting Economic Time Series," NBER Chapters,in: Economic Analysis of the Digital Economy, pages 119-135 National Bureau of Economic Research, Inc.
    4. Jennifer Castle & David Hendry & Jurgen A. Doornik, 2010. "Evaluating Automatic Model Selection," Economics Series Working Papers 474, University of Oxford, Department of Economics.

    More about this item

    Keywords

    Model selection algorithms; Information Criteria; General-to-Specific modeling; Bayesian Model Averaging; Portfolio Models; AIC; SIC; AICc; SICc; Monte Carlo Analysis; Autometrics;

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:09/13. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Albert Yee). General contact details of provider: http://edirc.repec.org/data/decannz.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.