IDEAS home Printed from
   My bibliography  Save this article

Identifying Latent Structures in Panel Data


  • Liangjun Su
  • Zhentao Shi
  • Peter C. B. Phillips


This paper provides a novel mechanism for identifying and estimating latent group structures in panel data using penalized techniques. We consider both linear and nonlinear models where the regression coefficients are heterogeneous across groups but homogeneous within a group and the group membership is unknown. Two approaches are considered—penalized profile likelihood (PPL) estimation for the general nonlinear models without endogenous regressors, and penalized GMM (PGMM) estimation for linear models with endogeneity. In both cases, we develop a new variant of Lasso called classifier‐Lasso (C‐Lasso) that serves to shrink individual coefficients to the unknown group‐specific coefficients. C‐Lasso achieves simultaneous classification and consistent estimation in a single step and the classification exhibits the desirable property of uniform consistency. For PPL estimation, C‐Lasso also achieves the oracle property so that group‐specific parameter estimators are asymptotically equivalent to infeasible estimators that use individual group identity information. For PGMM estimation, the oracle property of C‐Lasso is preserved in some special cases. Simulations demonstrate good finite‐sample performance of the approach in both classification and estimation. Empirical applications to both linear and nonlinear models are presented.

Suggested Citation

  • Liangjun Su & Zhentao Shi & Peter C. B. Phillips, 2016. "Identifying Latent Structures in Panel Data," Econometrica, Econometric Society, vol. 84, pages 2215-2264, November.
  • Handle: RePEc:wly:emetrp:v:84:y:2016:i::p:2215-2264

    Download full text from publisher

    File URL:
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    1. Su, Liangjun & Chen, Qihui, 2013. "Testing Homogeneity In Panel Data Models With Interactive Fixed Effects," Econometric Theory, Cambridge University Press, vol. 29(6), pages 1079-1135, December.
    2. Hahn, Jinyong & Kuersteiner, Guido, 2011. "Bias Reduction For Dynamic Nonlinear Panel Models With Fixed Effects," Econometric Theory, Cambridge University Press, vol. 27(6), pages 1152-1191, December.
    3. B. Prakasa Rao, 2009. "Conditional independence, conditional mixing and conditional association," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 61(2), pages 441-460, June.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Feng, Guohua & Gao, Jiti & Peng, Bin & Zhang, Xiaohui, 2017. "A varying-coefficient panel data model with fixed effects: Theory and an application to US commercial banks," Journal of Econometrics, Elsevier, vol. 196(1), pages 68-82.
    2. Wang, Wuyi & Phillips, Peter C.B. & Su, Liangjun, 2019. "The heterogeneous effects of the minimum wage on employment across states," Economics Letters, Elsevier, vol. 174(C), pages 179-185.
    3. repec:eee:econom:v:212:y:2019:i:2:p:451-475 is not listed on IDEAS
    4. repec:spt:stecon:v:7:y:2018:i:3:f:7_3_3 is not listed on IDEAS
    5. Jiti Gao & Kai Xia, 2017. "Heterogeneous panel data models with cross-sectional dependence," Monash Econometrics and Business Statistics Working Papers 16/17, Monash University, Department of Econometrics and Business Statistics.
    6. repec:eee:econom:v:206:y:2018:i:2:p:554-573 is not listed on IDEAS
    7. repec:eee:econom:v:209:y:2019:i:2:p:185-207 is not listed on IDEAS
    8. Ryo Okui & Takahide Yanagi, 2018. "Kernel Estimation for Panel Data with Heterogeneous Dynamics," Papers 1802.08825,, revised May 2019.
    9. Okui, Ryo & Yanagi, Takahide, 2019. "Panel data analysis with heterogeneous dynamics," Journal of Econometrics, Elsevier, vol. 212(2), pages 451-475.
    10. repec:eee:appene:v:210:y:2018:i:c:p:673-684 is not listed on IDEAS
    11. Kong, Jianning & Phillips, Peter C.B. & Sul, Donggyu, 2019. "Weak σ-convergence: Theory and applications," Journal of Econometrics, Elsevier, vol. 209(2), pages 185-207.
    12. Michael Vogt & Oliver Linton, 2015. "Classification of nonparametric regression functions in heterogeneous panels," CeMMAP working papers CWP06/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    13. Ayden Higgins & Federico Martellosio, 2019. "Shrinkage Estimation of Network Spillovers with Factor Structured Errors," Papers 1909.02823,
    14. Qian, Junhui & Su, Liangjun, 2016. "Shrinkage estimation of common breaks in panel data models via adaptive group fused Lasso," Journal of Econometrics, Elsevier, vol. 191(1), pages 86-109.
    15. Ryo Okui & Wendun Wang, 2018. "Heterogeneous structural breaks in panel data models," Papers 1801.04672,, revised Nov 2018.
    16. Jia Chen, 2018. "Estimating Latent Group Structure in Time-Varying Coefficient Panel Data Models," Discussion Papers 18/15, Department of Economics, University of York.
    17. repec:bla:jorssb:v:79:y:2017:i:1:p:5-27 is not listed on IDEAS

    More about this item

    JEL classification:

    • C33 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Models with Panel Data; Spatio-temporal Models
    • C36 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Instrumental Variables (IV) Estimation
    • C38 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Classification Methdos; Cluster Analysis; Principal Components; Factor Analysis
    • C51 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Construction and Estimation


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:emetrp:v:84:y:2016:i::p:2215-2264. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley Content Delivery). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.