IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v170y2007i4p1035-1059.html
   My bibliography  Save this article

Avoiding ‘data snooping’ in multilevel and mixed effects models

Author

Listed:
  • David Afshartous
  • Michael Wolf

Abstract

Summary. Multilevel or mixed effects models are commonly applied to hierarchical data. The level 2 residuals, which are otherwise known as random effects, are often of both substantive and diagnostic interest. Substantively, they are frequently used for institutional comparisons or rankings. Diagnostically, they are used to assess the model assumptions at the group level. Inference on the level 2 residuals, however, typically does not account for ‘data snooping’, i.e. for the harmful effects of carrying out a multitude of hypothesis tests at the same time. We provide a very general framework that encompasses both of the following inference problems: inference on the ‘absolute’ level 2 residuals to determine which are significantly different from 0, and inference on any prespecified number of pairwise comparisons. Thus, the user has the choice of testing the comparisons of interest. As our methods are flexible with respect to the estimation method that is invoked, the user may choose the desired estimation method accordingly. We demonstrate the methods with the London education authority data, the wafer data and the National Educational Longitudinal Study data.

Suggested Citation

  • David Afshartous & Michael Wolf, 2007. "Avoiding ‘data snooping’ in multilevel and mixed effects models," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 170(4), pages 1035-1059, October.
  • Handle: RePEc:bla:jorssa:v:170:y:2007:i:4:p:1035-1059
    DOI: 10.1111/j.1467-985X.2007.00494.x
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/j.1467-985X.2007.00494.x
    Download Restriction: no

    File URL: https://libkey.io/10.1111/j.1467-985X.2007.00494.x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    2. Harvey Goldstein & Michael J. R. Healy, 1995. "The Graphical Presentation of a Collection of Means," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 158(1), pages 175-177, January.
    3. James R. Carpenter & Harvey Goldstein & Jon Rasbash, 2003. "A novel bootstrap procedure for assessing the relationship between class size and achievement," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 52(4), pages 431-443, October.
    4. Joseph P. Romano & Michael Wolf, "undated". "Control of Generalized Error Rates in Multiple Testing," IEW - Working Papers 245, Institute for Empirical Research in Economics - University of Zurich.
    5. Valerie S. L. Williams & Lyle V. Jones & John W. Tukey, 1999. "Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement," Journal of Educational and Behavioral Statistics, , vol. 24(1), pages 42-69, March.
    6. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "multiple testing," The New Palgrave Dictionary of Economics,, Palgrave Macmillan.
    7. Harvey Goldstein & David J. Spiegelhalter, 1996. "League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 159(3), pages 385-409, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anders Skrondal & Sophia Rabe‐Hesketh, 2009. "Prediction in multilevel generalized linear models," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(3), pages 659-687, June.
    2. George Leckie & Harvey Goldstein, 2009. "The limitations of using school league tables to inform school choice," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(4), pages 835-851, October.
    3. Afshartous, David & Preston, Richard A., 2010. "Confidence intervals for dependent data: Equating non-overlap with statistical significance," Computational Statistics & Data Analysis, Elsevier, vol. 54(10), pages 2296-2305, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Romano, Joseph P. & Shaikh, Azeem M. & Wolf, Michael, 2008. "Formalized Data Snooping Based On Generalized Error Rates," Econometric Theory, Cambridge University Press, vol. 24(2), pages 404-447, April.
    2. Magne Mogstad & Joseph P Romano & Azeem M Shaikh & Daniel Wilhelm, 2024. "Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 91(1), pages 476-518.
    3. Giuseppe Cavaliere & Dimitris N. Politis & Anders Rahbek & Paul Doukhan & Gabriel Lang & Anne Leucht & Michael H. Neumann, 2015. "Recent developments in bootstrap methods for dependent data," Journal of Time Series Analysis, Wiley Blackwell, vol. 36(3), pages 290-314, May.
    4. Tania Singer & Ernst Fehr, 2005. "The Neuroeconomics of Mind Reading and Empathy," American Economic Review, American Economic Association, vol. 95(2), pages 340-345, May.
    5. Heath, Davidson & Ringgenberg, Matthew C. & Samadi, Mehrdad & Werner, Ingrid M., 2019. "Reusing Natural Experiments," Working Paper Series 2019-21, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    6. Joseph Romano & Azeem Shaikh & Michael Wolf, 2008. "Control of the false discovery rate under dependence using the bootstrap and subsampling," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(3), pages 417-442, November.
    7. Dichtl, Hubert & Drobetz, Wolfgang & Neuhierl, Andreas & Wendt, Viktoria-Sophie, 2021. "Data snooping in equity premium prediction," International Journal of Forecasting, Elsevier, vol. 37(1), pages 72-94.
    8. Armin Falk & Ernst Fehr & Christian Zehnder, "undated". "The Behavioral Effects of Minimum Wages," IEW - Working Papers 247, Institute for Empirical Research in Economics - University of Zurich.
    9. Kathleen Carey, 2000. "A multilevel modelling approach to analysis of patient costs under managed care," Health Economics, John Wiley & Sons, Ltd., vol. 9(5), pages 435-446, July.
    10. George Leckie & Harvey Goldstein, 2009. "The limitations of using school league tables to inform school choice," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(4), pages 835-851, October.
    11. Sergei Bazylik & Magne Mogstad & Joseph P. Romano & Azeem Shaikh & Daniel Wilhelm, 2021. "Finite- and Large-Sample Inference for Ranks using Multinomial Data with an Application to Ranking Political Parties," NBER Working Papers 29519, National Bureau of Economic Research, Inc.
    12. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    13. Isabella Sulis & Mariano Porcu, 2015. "Assessing Divergences in Mathematics and Reading Achievement in Italian Primary Schools: A Proposal of Adjusted Indicators of School Effectiveness," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 122(2), pages 607-634, June.
    14. Isabella Sulis & Mariano Porcu & Vincenza Capursi, 2019. "On the Use of Student Evaluation of Teaching: A Longitudinal Analysis Combining Measurement Issues and Implications of the Exercise," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 142(3), pages 1305-1331, April.
    15. Hsu, Po-Hsuan & Han, Qiheng & Wu, Wensheng & Cao, Zhiguang, 2018. "Asset allocation strategies, data snooping, and the 1 / N rule," Journal of Banking & Finance, Elsevier, vol. 97(C), pages 257-269.
    16. Laurent Barras & Olivier Scaillet & Russ Wermers, 2010. "False Discoveries in Mutual Fund Performance: Measuring Luck in Estimated Alphas," Journal of Finance, American Finance Association, vol. 65(1), pages 179-216, February.
    17. Nicholas Tibor Longford, 2016. "Decision Theory Applied to Selecting the Winners, Ranking, and Classification," Journal of Educational and Behavioral Statistics, , vol. 41(4), pages 420-442, August.
    18. Hassler Uwe & Werkmann Verena, 2014. "Multiple Comparisons and Joint Significance in Panel Unit Root Testing with Evidence on International Interest Rate Linkage," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 234(1), pages 23-43, February.
    19. Mathur, Maya B & VanderWeele, Tyler, 2018. "Statistical methods for evidence synthesis," Thesis Commons kd6ja, Center for Open Science.
    20. Michael Wolf & Dan Wunderli, 2012. "Bootstrap joint prediction regions," ECON - Working Papers 064, Department of Economics - University of Zurich, revised May 2013.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:170:y:2007:i:4:p:1035-1059. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.