IDEAS home Printed from https://ideas.repec.org/a/spr/stpapr/v59y2018i2d10.1007_s00362-016-0781-8.html
   My bibliography  Save this article

Sparse factor regression via penalized maximum likelihood estimation

Author

Listed:
  • Kei Hirose

    (Kyushu University)

  • Miyuki Imada

    (NTT Network Innovation Laboratories)

Abstract

In factor regression model, the maximum likelihood estimation suffers from three disadvantages: (i) the maximum likelihood estimates are unavailable when the number of variables exceeds the number of observations, (ii) the rotation technique based on maximum likelihood estimates produces an insufficiently sparse loading matrix, and (iii) multicollinearity can occur when the estimates of unique variances (specific variances) are small because the regression coefficients are sensitive to the inverse of unique variances. To handle these problems, we propose a penalized maximum likelihood procedure. Specifically, we impose a lasso-type penalty on the factor loadings to improve the sparseness of the solution. We also introduce a penalty on unique variances, which (given the factor scores) corresponds to the ridge penalty on the regression coefficient. Theoretical properties from a prediction viewpoint of our procedure are discussed. The effectiveness of the procedure is investigated through Monte Carlo simulations. The utility of our procedure is demonstrated on real data collected by an online questionnaire.

Suggested Citation

  • Kei Hirose & Miyuki Imada, 2018. "Sparse factor regression via penalized maximum likelihood estimation," Statistical Papers, Springer, vol. 59(2), pages 633-662, June.
    • Handle: RePEc:spr:stpapr:v:59:y:2018:i:2:d:10.1007_s00362-016-0781-8
    DOI: 10.1007/s00362-016-0781-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00362-016-0781-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00362-016-0781-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Hyonho Chun & Sündüz Keleş, 2010. "Sparse partial least squares regression for simultaneous dimension reduction and variable selection," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(1), pages 3-25, January.
    2. James Martin & Roderick McDonald, 1975. "Bayesian estimation in unrestricted factor analysis: A treatment for heywood cases," Psychometrika, Springer;The Psychometric Society, vol. 40(4), pages 505-517, December.
    3. Otto Driel, 1978. "On various causes of improper solutions in maximum likelihood factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 43(2), pages 225-243, June.
    4. Carvalho, Carlos M. & Chang, Jeffrey & Lucas, Joseph E. & Nevins, Joseph R. & Wang, Quanli & West, Mike, 2008. "High-Dimensional Sparse Factor Modeling: Applications in Gene Expression Genomics," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1438-1456.
    5. Zou, Hui, 2006. "The Adaptive Lasso and Its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1418-1429, December.
    6. Henry Kaiser, 1958. "The varimax criterion for analytic rotation in factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 23(3), pages 187-200, September.
    7. Kawano, Shuichi & Fujisawa, Hironori & Takada, Toyoyuki & Shiroishi, Toshihiko, 2015. "Sparse principal component regression with adaptive loading," Computational Statistics & Data Analysis, Elsevier, vol. 89(C), pages 192-203.
    8. Friedman, Jerome H. & Hastie, Trevor & Tibshirani, Rob, 2010. "Regularization Paths for Generalized Linear Models via Coordinate Descent," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 33(i01).
    9. Ulf Olsson, 1979. "Maximum likelihood estimation of the polychoric correlation coefficient," Psychometrika, Springer;The Psychometric Society, vol. 44(4), pages 443-460, December.
    10. Hirotugu Akaike, 1987. "Factor analysis and AIC," Psychometrika, Springer;The Psychometric Society, vol. 52(3), pages 317-332, September.
    11. Mazumder, Rahul & Friedman, Jerome H. & Hastie, Trevor, 2011. "SparseNet: Coordinate Descent With Nonconvex Penalties," Journal of the American Statistical Association, American Statistical Association, vol. 106(495), pages 1125-1138.
    12. Fuchun Huang, 2003. "Prediction Error Property of the Lasso Estimator and its Generalization," Australian & New Zealand Journal of Statistics, Australian Statistical Publishing Association Inc., vol. 45(2), pages 217-228, June.
    13. Donald Rubin & Dorothy Thayer, 1982. "EM algorithms for ML factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 47(1), pages 69-76, March.
    14. James Anderson & David Gerbing, 1984. "The effect of sampling error on convergence, improper solutions, and goodness-of-fit indices for maximum likelihood confirmatory factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 49(2), pages 155-173, June.
    15. Hui Zou & Trevor Hastie, 2005. "Addendum: Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(5), pages 768-768, November.
    16. Hui Zou & Trevor Hastie, 2005. "Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(2), pages 301-320, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shaoxin Wang & Hu Yang & Chaoli Yao, 2019. "On the penalized maximum likelihood estimation of high-dimensional approximate factor model," Computational Statistics, Springer, vol. 34(2), pages 819-846, June.
    2. Frederik Banis & Henrik Madsen & Niels K. Poulsen & Daniela Guericke, 2020. "Prosumer Response Estimation Using SINDyc in Conjunction with Markov-Chain Monte-Carlo Sampling," Energies, MDPI, vol. 13(12), pages 1-16, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kawano, Shuichi & Fujisawa, Hironori & Takada, Toyoyuki & Shiroishi, Toshihiko, 2015. "Sparse principal component regression with adaptive loading," Computational Statistics & Data Analysis, Elsevier, vol. 89(C), pages 192-203.
    2. Jin, Shaobo & Moustaki, Irini & Yang-Wallentin, Fan, 2018. "Approximated penalized maximum likelihood for exploratory factor analysis: an orthogonal case," LSE Research Online Documents on Economics 88118, London School of Economics and Political Science, LSE Library.
    3. Shaobo Jin & Irini Moustaki & Fan Yang-Wallentin, 2018. "Approximated Penalized Maximum Likelihood for Exploratory Factor Analysis: An Orthogonal Case," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 628-649, September.
    4. Bai, Ray & Ghosh, Malay, 2018. "High-dimensional multivariate posterior consistency under global–local shrinkage priors," Journal of Multivariate Analysis, Elsevier, vol. 167(C), pages 157-170.
    5. Shuichi Kawano, 2021. "Sparse principal component regression via singular value decomposition approach," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 15(3), pages 795-823, September.
    6. Hirose, Kei & Tateishi, Shohei & Konishi, Sadanori, 2013. "Tuning parameter selection in sparse regression modeling," Computational Statistics & Data Analysis, Elsevier, vol. 59(C), pages 28-40.
    7. Kawano, Shuichi & Fujisawa, Hironori & Takada, Toyoyuki & Shiroishi, Toshihiko, 2018. "Sparse principal component regression for generalized linear models," Computational Statistics & Data Analysis, Elsevier, vol. 124(C), pages 180-196.
    8. Yoonsuh Jung, 2018. "Multiple predicting K-fold cross-validation for model selection," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 30(1), pages 197-215, January.
    9. Tutz, Gerhard & Pößnecker, Wolfgang & Uhlmann, Lorenz, 2015. "Variable selection in general multinomial logit models," Computational Statistics & Data Analysis, Elsevier, vol. 82(C), pages 207-222.
    10. Margherita Giuzio, 2017. "Genetic algorithm versus classical methods in sparse index tracking," Decisions in Economics and Finance, Springer;Associazione per la Matematica, vol. 40(1), pages 243-256, November.
    11. Mkhadri, Abdallah & Ouhourane, Mohamed, 2013. "An extended variable inclusion and shrinkage algorithm for correlated variables," Computational Statistics & Data Analysis, Elsevier, vol. 57(1), pages 631-644.
    12. Christopher J Greenwood & George J Youssef & Primrose Letcher & Jacqui A Macdonald & Lauryn J Hagg & Ann Sanson & Jenn Mcintosh & Delyse M Hutchinson & John W Toumbourou & Matthew Fuller-Tyszkiewicz &, 2020. "A comparison of penalised regression methods for informing the selection of predictive markers," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-14, November.
    13. Mostafa Rezaei & Ivor Cribben & Michele Samorani, 2021. "A clustering-based feature selection method for automatically generated relational attributes," Annals of Operations Research, Springer, vol. 303(1), pages 233-263, August.
    14. Camila Epprecht & Dominique Guegan & Álvaro Veiga & Joel Correa da Rosa, 2017. "Variable selection and forecasting via automated methods for linear models: LASSO/adaLASSO and Autometrics," Post-Print halshs-00917797, HAL.
    15. Peter Bühlmann & Jacopo Mandozzi, 2014. "High-dimensional variable screening and bias in subsequent inference, with an empirical comparison," Computational Statistics, Springer, vol. 29(3), pages 407-430, June.
    16. Peter Martey Addo & Dominique Guegan & Bertrand Hassani, 2018. "Credit Risk Analysis Using Machine and Deep Learning Models," Risks, MDPI, vol. 6(2), pages 1-20, April.
    17. Capanu, Marinela & Giurcanu, Mihai & Begg, Colin B. & Gönen, Mithat, 2023. "Subsampling based variable selection for generalized linear models," Computational Statistics & Data Analysis, Elsevier, vol. 184(C).
    18. Tomáš Plíhal, 2021. "Scheduled macroeconomic news announcements and Forex volatility forecasting," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 40(8), pages 1379-1397, December.
    19. Loann David Denis Desboulets, 2018. "A Review on Variable Selection in Regression Analysis," Econometrics, MDPI, vol. 6(4), pages 1-27, November.
    20. Zeyu Bian & Erica E. M. Moodie & Susan M. Shortreed & Sahir Bhatnagar, 2023. "Variable selection in regression‐based estimation of dynamic treatment regimes," Biometrics, The International Biometric Society, vol. 79(2), pages 988-999, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:stpapr:v:59:y:2018:i:2:d:10.1007_s00362-016-0781-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.