Measuring Test Measurement Error: A General Approach
AbstractTest-based accountability including value-added assessments and experimental and quasi-experimental research in education rely on achievement tests to measure student skills and knowledge. Yet we know little regarding important properties of these tests, an important example being the extent of test measurement error and its implications for educational policy and practice. While test vendors provide estimates of split-test reliability, these measures do not account for potentially important day-to-day differences in student performance. We show there is a credible, low-cost approach for estimating the total test measurement error that can be applied when one or more cohorts of students take three or more tests in the subject of interest (e.g., state assessments in three consecutive grades). Our method generalizes the test-retest framework allowing for either growth or decay in knowledge and skills between tests as well as variation in the degree of measurement error across tests. The approach maintains relatively unrestrictive, testable assumptions regarding the structure of student achievement growth. Estimation only requires descriptive statistics (e.g., correlations) for the tests. When student-level test-score data are available, the extent and pattern of measurement error heteroskedasticity also can be estimated. Utilizing math and ELA test data from New York City, we estimate the overall extent of test measurement error is more than twice as large as that reported by the test vendor and demonstrate how using estimates of the total measurement error and the degree of heteroskedasticity along with observed scores can yield meaningful improvements in the precision of student achievement and achievement-gain estimates.
Download InfoIf you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.
As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.
Bibliographic InfoPaper provided by National Bureau of Economic Research, Inc in its series NBER Working Papers with number 18010.
Date of creation: Apr 2012
Date of revision:
Contact details of provider:
Postal: National Bureau of Economic Research, 1050 Massachusetts Avenue Cambridge, MA 02138, U.S.A.
Web page: http://www.nber.org
More information through EDIRC
Find related papers by JEL classification:
- I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
This paper has been announced in the following NEP Reports:
- NEP-ALL-2012-05-02 (All new papers)
- NEP-ECM-2012-05-02 (Econometrics)
- NEP-EDU-2012-05-02 (Education)
- NEP-URE-2012-05-02 (Urban & Real Estate Economics)
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:
- Daniel Aaronson & Lisa Barrow & William Sander, 2002.
"Teachers and student achievement in the Chicago public high schools,"
Working Paper Series
WP-02-28, Federal Reserve Bank of Chicago.
- Daniel Aaronson & Lisa Barrow & William Sander, 2007. "Teachers and Student Achievement in the Chicago Public High Schools," Journal of Labor Economics, University of Chicago Press, vol. 25, pages 95-135.
- Joseph G. Altonji & Lewis M. Segal, 1994.
"Small Sample Bias in GMM Estimation of Covariance Structures,"
NBER Technical Working Papers
0156, National Bureau of Economic Research, Inc.
- Altonji, Joseph G & Segal, Lewis M, 1996. "Small-Sample Bias in GMM Estimation of Covariance Structures," Journal of Business & Economic Statistics, American Statistical Association, vol. 14(3), pages 353-66, July.
- Joseph G. Altonji & Lewis M. Segal, 1994. "Small sample bias in GMM estimation of covariance structures," Working Paper Series, Macroeconomic Issues 94-8, Federal Reserve Bank of Chicago.
- Dan Goldhaber & Emily Anthony, 2007. "Can Teacher Quality Be Effectively Assessed? National Board Certification as a Signal of Effective Teaching," The Review of Economics and Statistics, MIT Press, vol. 89(1), pages 134-150, February.
- Cory Koedel & Julian Betts, 2007. "Re-Examining the Role of Teacher Quality In the Educational Production Function," Working Papers 0708, Department of Economics, University of Missouri.
- Petra E. Todd & Kenneth I. Wolpin, 2003. "On The Specification and Estimation of The Production Function for Cognitive Achievement," Economic Journal, Royal Economic Society, vol. 113(485), pages F3-F33, February.
- Eric Parsons, 2014. "The Impact of Attending Low-Achieving Schools on High-Performing Student Outcomes," Working Papers 1407, Department of Economics, University of Missouri.
- Cory Koedel & Jiaxi Li, 2014. "The Efficiency Implications of Using Proportional Evaluations to Shape the Teaching Workforce," Working Papers 1402, Department of Economics, University of Missouri.
- Timothy N. Bond & Kevin Lang, 2013. "The Black-White Education-Scaled Test-Score Gap in Grades K-7," NBER Working Papers 19243, National Bureau of Economic Research, Inc.
- Eric Parsons, 2013. "The Icarus Syndrome: Why Do Some High Flyers Soar While Others Fall?," Working Papers 1308, Department of Economics, University of Missouri.
- Jason A. Grissom & Demetra Kalogrides & Susanna Loeb, 2012. "Using Student Test Scores to Measure Principal Performance," NBER Working Papers 18568, National Bureau of Economic Research, Inc.
For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ().
If references are entirely missing, you can add them using this form.