Statistical Power in Cross-Sectional Multilevel Experiments in Education

This chapter focuses on a priori statistical power analysis in multilevel field experiments. Specifically, statistical power methods in two- and three-level cluster and block randomized designs are outlined. Altogether five experimental designs (two cluster and three block randomized) are defined. The sample size at the top level (e.g., schools) has a stronger impact on power than sample sizes at lower levels (e.g., classrooms, students) regardless of the type of the design. The magnitude of the treatment effect is positively related to power, and as the magnitude of the effect size increases, power increases as well. Clustering captured through intraclass correlations is inversely related to power, and thus decreases in clustering correspond to increases in power. Covariates that explain a considerable amount of the variance (e.g., ≥50%) at different levels increase power. In cluster designs, top-level covariates have a stronger influence on power compared to lower-level covariates (assuming cluster-mean centering). Power is typically larger in block randomized designs compared to cluster randomized designs. The two most powerful of the five designs seem to be the two-level block randomized design and the three-level block randomized design where the treatment is assigned at the bottom level.