IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/71670.html
   My bibliography  Save this paper

Model selection consistency from the perspective of generalization ability and VC theory with an application to Lasso

Author

Listed:
  • Xu, Ning
  • Hong, Jian
  • Fisher, Timothy

Abstract

Model selection is difficult to analyse yet theoretically and empirically important, especially for high-dimensional data analysis. Recently the least absolute shrinkage and selection operator (Lasso) has been applied in the statistical and econometric literature. Consis- tency of Lasso has been established under various conditions, some of which are difficult to verify in practice. In this paper, we study model selection from the perspective of generalization ability, under the framework of structural risk minimization (SRM) and Vapnik-Chervonenkis (VC) theory. The approach emphasizes the balance between the in-sample and out-of-sample fit, which can be achieved by using cross-validation to select a penalty on model complexity. We show that an exact relationship exists between the generalization ability of a model and model selection consistency. By implementing SRM and the VC inequality, we show that Lasso is L2-consistent for model selection under assumptions similar to those imposed on OLS. Furthermore, we derive a probabilistic bound for the distance between the penalized extremum estimator and the extremum estimator without penalty, which is dominated by overfitting. We also propose a new measurement of overfitting, GR2, based on generalization ability, that converges to zero if model selection is consistent. Using simulations, we demonstrate that the proposed CV-Lasso algorithm performs well in terms of model selection and overfitting control.

Suggested Citation

  • Xu, Ning & Hong, Jian & Fisher, Timothy, 2016. "Model selection consistency from the perspective of generalization ability and VC theory with an application to Lasso," MPRA Paper 71670, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:71670
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/71670/1/MPRA_paper_71670.pdf
    File Function: original version
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Zou, Hui, 2006. "The Adaptive Lasso and Its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1418-1429, December.
    2. Sebastiano Manzan, 2015. "Forecasting the Distribution of Economic Variables in a Data-Rich Environment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 33(1), pages 144-164, January.
    3. Chatterjee, A. & Gupta, S. & Lahiri, S.N., 2015. "On the residual empirical process based on the ALASSO in high dimensions and its functional oracle property," Journal of Econometrics, Elsevier, vol. 186(2), pages 317-324.
    4. B. Pistoresi & F. Salsano & D. Ferrari, 2011. "Political institutions and central bank independence revisited," Applied Economics Letters, Taylor & Francis Journals, vol. 18(7), pages 679-682.
    5. Ulrike Schneider & Martin Wagner, 2012. "Catching Growth Determinants with the Adaptive Lasso," German Economic Review, Verein für Socialpolitik, vol. 13(1), pages 71-85, February.
    6. Cheng, Xu & Liao, Zhipeng, 2015. "Select the valid and relevant moments: An information-based LASSO for GMM with many moments," Journal of Econometrics, Elsevier, vol. 186(2), pages 443-464.
    7. De Mol, Christine & Giannone, Domenico & Reichlin, Lucrezia, 2006. "Forecasting using a large number of predictors: is Bayesian regression a valid alternative to principal components?," Discussion Paper Series 1: Economic Studies 2006,32, Deutsche Bundesbank.
    8. A. Belloni & D. Chen & V. Chernozhukov & C. Hansen, 2012. "Sparse Models and Methods for Optimal Instruments With an Application to Eminent Domain," Econometrica, Econometric Society, vol. 80(6), pages 2369-2429, November.
    9. Bai, Jushan & Ng, Serena, 2008. "Forecasting economic time series using targeted predictors," Journal of Econometrics, Elsevier, vol. 146(2), pages 304-317, October.
    10. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2011. "Inference for high-dimensional sparse econometric models," CeMMAP working papers CWP41/11, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    11. Kock, Anders Bredahl & Callot, Laurent, 2015. "Oracle inequalities for high dimensional vector autoregressions," Journal of Econometrics, Elsevier, vol. 186(2), pages 325-344.
    12. Kim, Hyun Hak & Swanson, Norman R., 2014. "Forecasting financial and macroeconomic variables using data reduction methods: New empirical evidence," Journal of Econometrics, Elsevier, vol. 178(P2), pages 352-367.
    13. De Mol, Christine & Giannone, Domenico & Reichlin, Lucrezia, 2008. "Forecasting using a large number of predictors: Is Bayesian shrinkage a valid alternative to principal components?," Journal of Econometrics, Elsevier, vol. 146(2), pages 318-328, October.
    14. Caner, Mehmet, 2009. "Lasso-Type Gmm Estimator," Econometric Theory, Cambridge University Press, vol. 25(1), pages 270-290, February.
    15. Hal R. Varian, 2014. "Big Data: New Tricks for Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 28(2), pages 3-28, Spring.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    Model selection; VC theory; generalization ability; Lasso; high-dimensional data; structural risk minimization; cross validation.;

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:71670. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Joachim Winter). General contact details of provider: http://edirc.repec.org/data/vfmunde.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.