IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp7583.html
   My bibliography  Save this paper

Regression Analysis of Country Effects Using Multilevel Data: A Cautionary Tale

Author

Listed:
  • Bryan, Mark L.

    (University of Sheffield)

  • Jenkins, Stephen P.

    (London School of Economics)

Abstract

Cross-national differences in outcomes are often analysed using regression analysis of multilevel country datasets, examples of which include the ECHP, ESS, EU-SILC, EVS, ISSP, and SHARE. We review the regression methods applicable to this data structure, pointing out problems with the assessment of country-level factors that appear not to be widely appreciated, and illustrate our arguments using Monte-Carlo simulations and analysis of women's employment probabilities and work hours using EU SILC data. With large sample sizes of individuals within each country but a small number of countries, analysts can reliably estimate individual-level effects within each country but estimates of parameters summarising country effects are likely to be unreliable. Multilevel (hierarchical) modelling methods are commonly used in this context but they are no panacea.

Suggested Citation

  • Bryan, Mark L. & Jenkins, Stephen P., 2013. "Regression Analysis of Country Effects Using Multilevel Data: A Cautionary Tale," IZA Discussion Papers 7583, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp7583
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp7583.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Kedar, Orit, 2005. "How Diffusion of Power in Parliaments Affects Voter Choice," Political Analysis, Cambridge University Press, vol. 13(4), pages 410-429.
    2. James R. Carpenter & Harvey Goldstein & Jon Rasbash, 2003. "A novel bootstrap procedure for assessing the relationship between class size and achievement," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 52(4), pages 431-443, October.
    3. Beck, Nathaniel & Katz, Jonathan N., 1995. "What To Do (and Not to Do) with Time-Series Cross-Section Data," American Political Science Review, Cambridge University Press, vol. 89(3), pages 634-647, September.
    4. Cora J. M. Maas & Joop J. Hox, 2004. "Robustness issues in multilevel regression analysis," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 58(2), pages 127-137, May.
    5. Stephen G. Donald & Kevin Lang, 2007. "Inference with Difference-in-Differences and Other Panel Data," The Review of Economics and Statistics, MIT Press, vol. 89(2), pages 221-233, May.
    6. William Greene, 2004. "The behaviour of the maximum likelihood estimator of limited dependent variable models in the presence of fixed effects," Econometrics Journal, Royal Economic Society, vol. 7(1), pages 98-119, June.
    7. Borjas, George J. & Sueyoshi, Glenn T., 1994. "A two-stage estimator for probit models with structural group effects," Journal of Econometrics, Elsevier, vol. 64(1-2), pages 165-182.
    8. Austin Peter C, 2010. "Estimating Multilevel Logistic Regression Models When the Number of Clusters is Low: A Comparison of Different Statistical Software Procedures," The International Journal of Biostatistics, De Gruyter, vol. 6(1), pages 1-20, April.
    9. Bénédicte Vidaillet & V. d'Estaintot & P. Abécassis, 2005. "Introduction," Post-Print hal-00287137, HAL.
    10. Germáan Rodríguez & Noreen Goldman, 1995. "An Assessment of Estimation Procedures for Multilevel Models with Binary Responses," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(1), pages 73-89, January.
    11. Bowers, Jake & Drake, Katherine W., 2005. "EDA for HLM: Visualization when Probabilistic Inference Fails," Political Analysis, Cambridge University Press, vol. 13(4), pages 301-326.
    12. Huber, John D. & Kernell, Georgia & Leoni, Eduardo L., 2005. "Institutional Context, Cognitive Resources and Party Attachments Across Democracies," Political Analysis, Cambridge University Press, vol. 13(4), pages 365-386.
    13. Jeffrey M Wooldridge, 2010. "Econometric Analysis of Cross Section and Panel Data," MIT Press Books, The MIT Press, edition 2, volume 1, number 0262232588, December.
    14. Saxonhouse, Gary R, 1976. "Estimated Parameters as Dependent Variables," American Economic Review, American Economic Association, vol. 66(1), pages 178-183, March.
    15. David Card, 1995. "The Wage Curve: A Review," Journal of Economic Literature, American Economic Association, vol. 33(2), pages 285-299, June.
    16. Achen, Christopher H., 2005. "Two-Step Hierarchical Estimation: Beyond Regression Analysis," Political Analysis, Cambridge University Press, vol. 13(4), pages 447-456.
    17. A. Colin Cameron & Pravin K. Trivedi, 2010. "Microeconometrics Using Stata, Revised Edition," Stata Press books, StataCorp LP, number musr, March.
    18. Moulton, Brent R., 1986. "Random group effects and the precision of regression estimates," Journal of Econometrics, Elsevier, vol. 32(3), pages 385-397, August.
    19. Germán Rodríguez & Noreen Goldman, 2001. "Improved estimation procedures for multilevel models with binary response: a case‐study," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 164(2), pages 339-355.
    20. Daniel Stegmueller, 2013. "How Many Countries for Multilevel Modeling? A Comparison of Frequentist and Bayesian Approaches," American Journal of Political Science, John Wiley & Sons, vol. 57(3), pages 748-761, July.
    21. Bester, C. Alan & Conley, Timothy G. & Hansen, Christian B., 2011. "Inference with dependent data using cluster covariance estimators," Journal of Econometrics, Elsevier, vol. 165(2), pages 137-151.
    22. Kedar, Orit & Shively, W. Phillips, 2005. "Introduction to the Special Issue," Political Analysis, Cambridge University Press, vol. 13(4), pages 297-300.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hansen, Bruce E. & Lee, Seojeong, 2019. "Asymptotic theory for clustered samples," Journal of Econometrics, Elsevier, vol. 210(2), pages 268-290.
    2. Marisa Hidalgo-Hidalgo, 2019. "Childhood-Related Policies and Adult Poverty: Evidence from Some European Countries," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 144(1), pages 191-217, July.
    3. Aleksey Oshchepkov & Anna Shirokanova, 2020. "Multilevel Modeling For Economists: Why, When And How," HSE Working papers WP BRP 233/EC/2020, National Research University Higher School of Economics.
    4. Qian, Xuefeng & Tian, Bifei & Reed, W. Robert & Chen, Ziruo, 2018. "Searching for profit-shifting in China," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-25.
    5. Thomas Barrios & Rebecca Diamond & Guido W. Imbens & Michal Kolesár, 2012. "Clustering, Spatial Correlations, and Randomization Inference," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(498), pages 578-591, June.
    6. James G. MacKinnon & Matthew D. Webb, 2017. "Wild Bootstrap Inference for Wildly Different Cluster Sizes," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(2), pages 233-254, March.
    7. Andrew Bell & Malcolm Fairbrother & Kelvyn Jones, 2019. "Fixed and random effects models: making an informed choice," Quality & Quantity: International Journal of Methodology, Springer, vol. 53(2), pages 1051-1074, March.
    8. Entorf, Horst & Sattarova, Liliya, 2016. "The Analysis of Prison-Prisoner Data Using Cluster-Sample Econometrics: Prison Conditions and Prisoners' Assessments of the Future," IZA Discussion Papers 10209, Institute of Labor Economics (IZA).
    9. Ivan A. Canay & Andres Santos & Azeem M. Shaikh, 2018. "The wild bootstrap with a "small" number of "large" clusters," CeMMAP working papers CWP27/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    10. Rabe-Hesketh, Sophia & Skrondal, Anders & Pickles, Andrew, 2005. "Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects," Journal of Econometrics, Elsevier, vol. 128(2), pages 301-323, October.
    11. Joan Daouli & Michael Demoussis & Nicholas Giannakopoulos & Ioannis Laliotis, 2017. "The wage curve before and during the Greek economic crisis," Empirical Economics, Springer, vol. 52(1), pages 59-77, February.
    12. Herman G. van de Werfhorst & Anthony Heath, 2019. "Selectivity of Migration and the Educational Disadvantages of Second-Generation Immigrants in Ten Host Societies," European Journal of Population, Springer;European Association for Population Studies, vol. 35(2), pages 347-378, May.
    13. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    14. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    15. Jeffrey R. Vincent & Ismariah Ahmad & Norliyana Adnan & Walter B. Burwell & Subhrendu K. Pattanayak & Jie-Sheng Tan-Soo & Kyle Thomas, 2016. "Valuing Water Purification by Forests: An Analysis of Malaysian Panel Data," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 64(1), pages 59-80, May.
    16. Bell, Andrew & Jones, Kelvyn, 2015. "Explaining Fixed Effects: Random Effects Modeling of Time-Series Cross-Sectional and Panel Data," Political Science Research and Methods, Cambridge University Press, vol. 3(1), pages 133-153, January.
    17. Guido W. Imbens & Michal Kolesár, 2016. "Robust Standard Errors in Small Samples: Some Practical Advice," The Review of Economics and Statistics, MIT Press, vol. 98(4), pages 701-712, October.
    18. Anastasiou, Dimitrios & Bragoudakis, Zacharias & Giannoulakis, Stelios, 2021. "Perceived vs actual financial crisis and bank credit standards: Is there any indication of self-fulfilling prophecy?," Research in International Business and Finance, Elsevier, vol. 58(C).
    19. Jeffrey Milyo & David M. Primo & Matthew L. Jacobsmeier, 2006. "Estimating the Impact of State Policies and Institutions with Mixed-Level Data," Working Papers 0603, Department of Economics, University of Missouri.
    20. Hug, Simon & Spörri, Franziska, 2011. "Referendums, trust, and tax evasion," European Journal of Political Economy, Elsevier, vol. 27(1), pages 120-131, March.

    More about this item

    Keywords

    cross-national comparisons; multilevel modelling; country effects;
    All these keywords.

    JEL classification:

    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • O57 - Economic Development, Innovation, Technological Change, and Growth - - Economywide Country Studies - - - Comparative Studies of Countries

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp7583. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.