IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v38y2014i6p546-582.html
   My bibliography  Save this article

Reference Values of Within-District Intraclass Correlations of Academic Achievement by District Characteristics

Author

Listed:
  • E. C. Hedberg
  • Larry V. Hedges

Abstract

Background: Randomized experiments are often considered the strongest designs to study the impact of educational interventions. Perhaps the most prevalent class of designs used in large-scale education experiments is the cluster randomized design in which entire schools are assigned to treatments. In cluster randomized trials that assign schools to treatments within a set of school districts, the statistical power of the test for treatment effects depends on the within-district school-level intraclass correlation (ICC). Hedges and Hedberg (2014) recently computed within-district ICC values in 11 states using three-level models (students in schools in districts) that pooled results across all the districts within each state. Although values from these analyses are useful when working with a representative sample of districts, they may be misleading for other samples of districts because the magnitude of ICCs appears to be related to district size. To plan studies with small or nonrepresentative samples of districts, better information are needed about the relation of within-district school-level ICCs to district size. Objective: Our objective is to explore the relation between district size and within-district ICCs to provide reference values for math and reading achievement for Grades 3–8 by district size, poverty level, and urbanicity level. These values are not derived from pooling across all districts within a state as in previous work but are based on the direct calculation of within-district school-level ICCs for each school district. Research Design: We use mixed models to estimate over 7,000 district-specific ICCs for math and reading achievement in 11 states and for Grades 3–8. We then perform a random effects meta-analysis on the estimated within-district ICCs. Our analysis is performed by grade and subject for different strata designated by district size (number of schools), urbanicity, and poverty rates.

Suggested Citation

  • E. C. Hedberg & Larry V. Hedges, 2014. "Reference Values of Within-District Intraclass Correlations of Academic Achievement by District Characteristics," Evaluation Review, , vol. 38(6), pages 546-582, December.
    • Handle: RePEc:sae:evarev:v:38:y:2014:i:6:p:546-582
    DOI: 10.1177/0193841X14554212
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X14554212
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X14554212?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:mpr:mprres:7064 is not listed on IDEAS
    2. Eric C. Hedberg, 2011. "RDPOWER: Stata module to perform power calculations for random designs," Statistical Software Components S457260, Boston College Department of Economics, revised 12 Feb 2012.
    3. repec:mpr:mprres:6586 is not listed on IDEAS
    4. Roberto Agodini & Barbara Harris & Sally Atkins-Burnett & Sheila Heaviside & Timothy Novak & Robert Murphy, "undated". "Achievement Effects of Four Early Elementary School Math Curricula: Findings from First Graders in 39 Schools," Mathematica Policy Research Reports 467194cd2d6f4cbaba7e23745, Mathematica Policy Research.
    5. repec:mpr:mprres:5863 is not listed on IDEAS
    6. repec:mpr:mprres:6365 is not listed on IDEAS
    7. repec:mpr:mprres:6816 is not listed on IDEAS
    8. Larry V. Hedges & E. C. Hedberg, 2013. "Intraclass Correlations and Covariate Outcome Correlations for Planning Two- and Three-Level Cluster-Randomized Experiments in Education," Evaluation Review, , vol. 37(6), pages 445-489, December.
    9. Peter Z. Schochet, "undated". "Statistical Power for Random Assignment Evaluations of Education Programs," Mathematica Policy Research Reports 6749d31ad72d4acf988f7dce5, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Zachary Neal, 2021. "Does the neighbourhood matter for neighbourhood satisfaction? A meta-analysis," Urban Studies, Urban Studies Journal Limited, vol. 58(9), pages 1775-1791, July.
    2. Nathan M. VanHoudnos & Joel B. Greenhouse, 2016. "On the Hedges Correction for a t-Test," Journal of Educational and Behavioral Statistics, , vol. 41(4), pages 392-419, August.
    3. E. C. Hedberg, 2016. "Academic and Behavioral Design Parameters for Cluster Randomized Trials in Kindergarten," Evaluation Review, , vol. 40(4), pages 279-313, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Elizabeth Tipton & Robert B. Olsen, "undated". "Enhancing the Generalizability of Impact Studies in Education," Mathematica Policy Research Reports 35d5625333dc480aba9765b3b, Mathematica Policy Research.
    2. Jessaca Spybrook & Benjamin Kelcey, 2016. "Introduction to Three Special Issues on Design Parameter Values for Planning Cluster Randomized Trials in the Social Sciences," Evaluation Review, , vol. 40(6), pages 491-499, December.
    3. Ben Kelcey & Zuchao Shen & Jessaca Spybrook, 2016. "Intraclass Correlation Coefficients for Designing Cluster-Randomized Trials in Sub-Saharan Africa Education," Evaluation Review, , vol. 40(6), pages 500-525, December.
    4. World Bank, 2017. "Pre-Primary Education in Mongolia," World Bank Publications - Reports 26402, The World Bank Group.
    5. Nianbo Dong & Wendy M. Reinke & Keith C. Herman & Catherine P. Bradshaw & Desiree W. Murray, 2016. "Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes," Evaluation Review, , vol. 40(4), pages 334-377, August.
    6. repec:mpr:mprres:6286 is not listed on IDEAS
    7. Douglas H. Clements & Roberto Agodini & Barbara Harris, "undated". "Instructional Practices and Student Math Achievement: Correlations from a Study of Math Curricula (Issue Brief)," Mathematica Policy Research Reports 1e18ce07a82f42479daa643f7, Mathematica Policy Research.
    8. repec:mpr:mprres:7907 is not listed on IDEAS
    9. Robin Jacob & Marie-Andree Somers & Pei Zhu & Howard Bloom, 2016. "The Validity of the Comparative Interrupted Time Series Design for Evaluating the Effect of School-Level Interventions," Evaluation Review, , vol. 40(3), pages 167-198, June.
    10. Christopher Rhoads, 2017. "Coherent Power Analysis in Multilevel Studies Using Parameters From Surveys," Journal of Educational and Behavioral Statistics, , vol. 42(2), pages 166-194, April.
    11. Rebecca A. Maynard, 2006. "Presidential address: Evidence-based decision making: What will it take for the decision makers to care?," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 25(2), pages 249-265.
    12. Kenneth Fortson & Natalya Verbitsky-Savitz & Emma Kopa & Philip Gleason, 2012. "Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates," Mathematica Policy Research Reports 27f871b5b7b94f3a80278a593, Mathematica Policy Research.
    13. Daniel McNeish & Jeffrey R. Harring & Denis Dumas, 2023. "A multilevel structured latent curve model for disaggregating student and school contributions to learning," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 32(2), pages 545-575, June.
    14. Deborah Peikes & Stacy Dale & Eric Lundquist & Janice Genevro & David Meyers, 2011. "Building the Evidence Base for the Medical Home: What Sample and Sample Size Do Studies Need?," Mathematica Policy Research Reports 5814eb8219b24982af7f7536c, Mathematica Policy Research.
    15. Peter Z. Schochet, 2018. "Design-Based Estimators for Average Treatment Effects for Multi-Armed RCTs," Journal of Educational and Behavioral Statistics, , vol. 43(5), pages 568-593, October.
    16. Andrew P. Jaciw & Li Lin & Boya Ma, 2016. "An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials," Evaluation Review, , vol. 40(5), pages 410-443, October.
    17. repec:mpr:mprres:6094 is not listed on IDEAS
    18. John Deke, 2016. "Design and Analysis Considerations for Cluster Randomized Controlled Trials That Have a Small Number of Clusters," Evaluation Review, , vol. 40(5), pages 444-486, October.
    19. Loyalka, Prashant & Song, Yingquan & Wei, Jianguo & Zhong, Weiping & Rozelle, Scott, 2013. "Information, college decisions and financial aid: Evidence from a cluster-randomized controlled trial in China," Economics of Education Review, Elsevier, vol. 36(C), pages 26-40.
    20. Kimberly Boller & Sally Atkins-Burnett & Elizabeth M. Malone & Gail P. Baxter & Jerry West, "undated". "Compendium of Student, Teacher, and Classroom Measures Used in NCEE Evaluations of Educational Interventions, Volume I: Measures Selection Approaches and Compendium Development Methods," Mathematica Policy Research Reports 07a8a75ba5634b3a813e22bc6, Mathematica Policy Research.
    21. John Deke & Lisa Dragoset, "undated". "Statistical Power for Regression Discontinuity Designs in Education: Empirical Estimates of Design Effects Relative to Randomized Controlled Trials," Mathematica Policy Research Reports a4f1d03eb7bf427a8983d4736, Mathematica Policy Research.
    22. repec:mpr:mprres:7909 is not listed on IDEAS
    23. repec:mpr:mprres:6721 is not listed on IDEAS
    24. Roberto Agodini & Barbara Harris & Neil Seftor & Janine Remillard & Melissa Thomas, "undated". "After Two Years, Three Elementary Math Curricula Outperform a Fourth (Issue Brief)," Mathematica Policy Research Reports 3941891e2a37463f8d7a9683d, Mathematica Policy Research.
    25. Peter Z. Schochet & Hanley Chiang, "undated". "Technical Methods Report: Estimation and Identification of the Complier Average Causal Effect Parameter in Education RCTs," Mathematica Policy Research Reports 947d1823e3ff42208532a763d, Mathematica Policy Research.

    More about this item

    Keywords

    education; methodological development;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:38:y:2014:i:6:p:546-582. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.