IDEAS home Printed from
   My bibliography  Save this paper

Identifying Latent Structures in Panel Data




This paper provides a novel mechanism for identifying and estimating latent group structures in panel data using penalized regression techniques. We focus on linear models where the slope parameters are heterogeneous across groups but homogenous within a group and the group membership is unknown. Two approaches are considered -- penalized least squares (PLS) for models without endogenous regressors, and penalized GMM (PGMM) for models with endogeneity. In both cases we develop a new variant of Lasso called classifier-Lasso (C-Lasso) that serves to shrink individual coefficients to the unknown group-specific coefficients. C-Lasso achieves simultaneous classification and consistent estimation in a single step and the classification exhibits the desirable property of uniform consistency. For PLS estimation C-Lasso also achieves the oracle property so that group-specific parameter estimators are asymptotically equivalent to infeasible estimators that use individual group identity information. For PGMM estimation the oracle property of C-Lasso is preserved in some special cases. Simulations demonstrate good finite-sample performance of the approach both in classification and estimation. An empirical application investigating the determinants of cross-country savings rates finds two latent groups among 56 countries, providing empirical confirmation that higher savings rates go in hand with higher income growth.

Suggested Citation

  • Liangjun Su & Zhentao Shi & Peter C.B. Phillips, 2014. "Identifying Latent Structures in Panel Data," Cowles Foundation Discussion Papers 1965, Cowles Foundation for Research in Economics, Yale University.
  • Handle: RePEc:cwl:cwldpp:1965

    Download full text from publisher

    File URL:
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    1. Su, Liangjun & Chen, Qihui, 2013. "Testing Homogeneity In Panel Data Models With Interactive Fixed Effects," Econometric Theory, Cambridge University Press, vol. 29(6), pages 1079-1135, December.
    2. Hahn, Jinyong & Kuersteiner, Guido, 2011. "Bias Reduction For Dynamic Nonlinear Panel Models With Fixed Effects," Econometric Theory, Cambridge University Press, vol. 27(6), pages 1152-1191, December.
    3. B. Prakasa Rao, 2009. "Conditional independence, conditional mixing and conditional association," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 61(2), pages 441-460, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Su, Liangjun & Jin, Sainan & Zhang, Yonghui, 2015. "Specification test for panel data models with interactive fixed effects," Journal of Econometrics, Elsevier, vol. 186(1), pages 222-244.
    2. Lu, Xun & Su, Liangjun, 2016. "Shrinkage estimation of dynamic panel data models with interactive fixed effects," Journal of Econometrics, Elsevier, vol. 190(1), pages 148-175.
    3. Su, Liangjun & Ju, Gaosheng, 2018. "Identifying latent grouped patterns in panel data models with interactive fixed effects," Journal of Econometrics, Elsevier, vol. 206(2), pages 554-573.
    4. Yiren Wang & Liangjun Su & Yichong Zhang, 2022. "Low-rank Panel Quantile Regression: Estimation and Inference," Papers 2210.11062,
    5. Miao, Ke & Su, Liangjun & Wang, Wendun, 2020. "Panel threshold regressions with latent group structures," Journal of Econometrics, Elsevier, vol. 214(2), pages 451-481.
    6. Wang, Wuyi & Su, Liangjun, 2021. "Identifying latent group structures in nonlinear panels," Journal of Econometrics, Elsevier, vol. 220(2), pages 272-295.
    7. Hong, Shengjie & Su, Liangjun & Jiang, Tao, 2023. "Profile GMM estimation of panel data models with interactive fixed effects," Journal of Econometrics, Elsevier, vol. 235(2), pages 927-948.
    8. Schumann, Martin & Severini, Thomas A. & Tripathi, Gautam, 2023. "The role of score and information bias in panel data likelihoods," Journal of Econometrics, Elsevier, vol. 235(2), pages 1215-1238.
    9. Lu, Xun & Su, Liangjun, 2023. "Uniform inference in linear panel data models with two-dimensional heterogeneity," Journal of Econometrics, Elsevier, vol. 235(2), pages 694-719.
    10. Ivan Fernandez-Val & Martin Weidner, 2014. "Individual and time effects in nonlinear panel models with large N , T," CeMMAP working papers 32/14, Institute for Fiscal Studies.
    11. Wang, Yiren & Phillips, Peter C.B. & Su, Liangjun, 2024. "Panel data models with time-varying latent group structures," Journal of Econometrics, Elsevier, vol. 240(1).
    12. repec:hal:spmain:info:hdl:2441/dambferfb7dfprc9m052g20qh is not listed on IDEAS
    13. Qi Guo & Bruno Remillard & Anatoliy Swishchuk, 2020. "Multivariate General Compound Point Processes in Limit Order Books," Risks, MDPI, vol. 8(3), pages 1-20, September.
    14. Iván Fernández-Val & Martin Weidner, 2018. "Fixed Effects Estimation of Large-TPanel Data Models," Annual Review of Economics, Annual Reviews, vol. 10(1), pages 109-138, August.
    15. Manuel Arellano & Stéphane Bonhomme, 2009. "Robust Priors in Nonlinear Panel Data Models," Econometrica, Econometric Society, vol. 77(2), pages 489-536, March.
    16. Manuel Arellano & Stéphane Bonhomme & Micole De Vera & Laura Hospido & Siqi Wei, 2022. "Income risk inequality: Evidence from Spanish administrative records," Quantitative Economics, Econometric Society, vol. 13(4), pages 1747-1801, November.
    17. Wang, Fa, 2017. "Maximum likelihood estimation and inference for high dimensional nonlinear factor models with application to factor-augmented regressions," MPRA Paper 93484, University Library of Munich, Germany, revised 19 May 2019.
    18. Geert Dhaene & Koen Jochmans, 2015. "Split-panel Jackknife Estimation of Fixed-effect Models," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(3), pages 991-1030.
    19. Shiu, Ji-Liang & Hu, Yingyao, 2013. "Identification and estimation of nonlinear dynamic panel data models with unobserved covariates," Journal of Econometrics, Elsevier, vol. 175(2), pages 116-131.
    20. Andersen, Torben G. & Fusari, Nicola & Todorov, Viktor & Varneskov, Rasmus T., 2019. "Unified inference for nonlinear factor models from panels with fixed and large time span," Journal of Econometrics, Elsevier, vol. 212(1), pages 4-25.
    21. Antonio F. Galvao & Thomas Parker & Zhijie Xiao, 2024. "Bootstrap Inference for Panel Data Quantile Regression," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 42(2), pages 628-639, April.

    More about this item


    Classification; Cluster analysis; Convergence club; Dynamic panel; Group Lasso; High dimensionality; Oracle property; Panel structure model; Parameter heterogeneity; Penalized least squares; Penalized GMM;
    All these keywords.

    JEL classification:

    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models
    • C36 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Instrumental Variables (IV) Estimation
    • C38 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Classification Methdos; Cluster Analysis; Principal Components; Factor Analysis
    • C51 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Construction and Estimation

    NEP fields

    This paper has been announced in the following NEP Reports:


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cwl:cwldpp:1965. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Brittany Ladd (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.