IDEAS home Printed from https://ideas.repec.org/a/sae/anname/v589y2003i1p114-149.html
   My bibliography  Save this article

Why have Educational Evaluators Chosen Not to Do Randomized Experiments?

Author

Listed:
  • Thomas D. Cook

Abstract

This article analyzes the reasons that have been adduced within the community of educational evaluators for not doing randomized experiments. The objections vary in cogency. Those that have most substance are not insurmountable, however, and strategies are mentioned for dealing with them. However, the objections are serious enough, and the remedies partial enough, that it seems hardly warranted to call experiments the “gold standard†of causal inference. Yet even if they are not perfect in research practice, this article shows how they are logically and empirically superior to all currently known alternatives. The article particularly addresses the objection that school personnel will not accept experiments. It shows that hundreds of them have been done there by researchers with backgrounds in psychology and public health who study the prevention of unhealthy behaviors. But experiments are much rarer among researchers trained in education who study changing academic performance. Reasons are adduced for this difference in academic culture within school-based research.

Suggested Citation

  • Thomas D. Cook, 2003. "Why have Educational Evaluators Chosen Not to Do Randomized Experiments?," The ANNALS of the American Academy of Political and Social Science, , vol. 589(1), pages 114-149, September.
    • Handle: RePEc:sae:anname:v:589:y:2003:i:1:p:114-149
    DOI: 10.1177/0002716203254764
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0002716203254764
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0002716203254764?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    2. Alan B. Krueger, 1999. "Experimental Estimates of Education Production Functions," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(2), pages 497-532.
    3. repec:mpr:mprres:1795 is not listed on IDEAS
    4. Krueger, Alan B & Whitmore, Diane M, 2001. "The Effect of Attending a Small Class in the Early Grades on College-Test Taking and Middle School Test Results: Evidence from Project STAR," Economic Journal, Royal Economic Society, vol. 111(468), pages 1-28, January.
    5. Cecilia Elena Rouse, 1998. "Private School Vouchers and Student Achievement: An Evaluation of the Milwaukee Parental Choice Program," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 113(2), pages 553-602.
    6. Friedlander, Daniel & Robins, Philip K, 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods," American Economic Review, American Economic Association, vol. 85(4), pages 923-937, September.
    7. Steven Glazerman & Dan M. Levy & David Myers, "undated". "Nonexperimental Replications of Social Experiments: A Systematic Review," Mathematica Policy Research Reports 9e8a2edf60ef4fa89ad9a5acf, Mathematica Policy Research.
    8. Eric A. Hanushek, "undated". "The Evidence on Class Size," Wallis Working Papers WP10, University of Rochester - Wallis Institute of Political Economy.
    9. repec:mpr:mprres:1895 is not listed on IDEAS
    10. repec:mpr:mprres:3369 is not listed on IDEAS
    11. Mark Dynarski & Robert Wood, 1997. "Helping High-Risk Youths: Results from the Alternative Schools Demonstration Program," Mathematica Policy Research Reports b10f23da06064e5ca34f56a27, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Xi Chen & Yan Liu & Cheng Zhang, 2022. "Distinguishing Homophily from Peer Influence Through Network Representation Learning," INFORMS Journal on Computing, INFORMS, vol. 34(4), pages 1958-1969, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rosalind Levacic & Stephen Machin & David Reynolds & Anna Vignoles & James Walker, 2000. "The Relationship between Resource Allocation and Pupil Attainment: A Review," CEE Discussion Papers 0002, Centre for the Economics of Education, LSE.
    2. John Bishop & Ludger Wossmann, 2004. "Institutional Effects in a Simple Model of Educational Production," Education Economics, Taylor & Francis Journals, vol. 12(1), pages 17-38.
    3. Marie Connolly & Catherine Haeck, 2018. "Le lien entre la taille des classes et les compétences cognitives et non cognitives," CIRANO Project Reports 2018rp-18, CIRANO.
    4. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    5. Will Dobbie & Roland G. Fryer, Jr, 2009. "Are High Quality Schools Enough to Close the Achievement Gap? Evidence from a Social Experiment in Harlem," NBER Working Papers 15473, National Bureau of Economic Research, Inc.
    6. Cecilia Elena Rouse & Jane Hannaway & Dan Goldhaber & David Figlio, 2013. "Feeling the Florida Heat? How Low-Performing Schools Respond to Voucher and Accountability Pressure," American Economic Journal: Economic Policy, American Economic Association, vol. 5(2), pages 251-281, May.
    7. Wößmann, Ludger, 2001. "New Evidence on the Missing Resource-Performance Link in Education," Kiel Working Papers 1051, Kiel Institute for the World Economy (IfW Kiel).
    8. Margaret Stevens & Kathryn Graddy, 2003. "The Impact of School Inputs on Student Performance: An Empirical Study of Private Schools in the United Kingdom," Economics Series Working Papers 146, University of Oxford, Department of Economics.
    9. Maximilian Bach & Stephan Sievert, 2019. "Birth Cohort Size Variation and the Estimation of Class Size Effects," Discussion Papers of DIW Berlin 1817, DIW Berlin, German Institute for Economic Research.
    10. Thomas D. Cook, "undated". "Considering the Major Arguments Against Random Assignment: An Analysis of the Intellectual Culture Surrounding Evaluation in American Schools of Education," IPR working papers 99-2, Institute for Policy Resarch at Northwestern University.
    11. Kokkelenberg, Edward C. & Dillon, Michael & Christy, Sean M., 2008. "The effects of class size on student grades at a public university," Economics of Education Review, Elsevier, vol. 27(2), pages 221-233, April.
    12. Stevens, Margaret & Graddy, Kathryn, 2003. "The Impact of School Inputs on Student Performance: An Empirical Study of Private Schools in the UK," CEPR Discussion Papers 3776, C.E.P.R. Discussion Papers.
    13. Roland G. Fryer, 2011. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," NBER Working Papers 16850, National Bureau of Economic Research, Inc.
    14. Vivian C. Wong & Peter M. Steiner, 2018. "Designs of Empirical Evaluations of Nonexperimental Methods in Field Settings," Evaluation Review, , vol. 42(2), pages 176-213, April.
    15. Matthew M. Chingos & Kenneth A. Couch, 2013. "Class Size and Student Outcomes: Research and Policy Implications," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(2), pages 411-438, March.
    16. Alan B. Krueger, 2003. "Economic Considerations and Class Size," Economic Journal, Royal Economic Society, vol. 113(485), pages 34-63, February.
    17. Michael A. Boozer & Stephen E. Cacciola, 2001. "Inside the 'Black Box' of Project STAR: Estimation of Peer Effects Using Experimental Data," Working Papers 832, Economic Growth Center, Yale University.
    18. Justman, Moshe, 2018. "Randomized controlled trials informing public policy: Lessons from project STAR and class size reduction," European Journal of Political Economy, Elsevier, vol. 54(C), pages 167-174.
    19. Elizabeth Ty Wilde & Robinson Hollister, 2007. "How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 26(3), pages 455-477.
    20. Moshe Justman, 2016. "Economic Research and Education Policy: Project STAR and Class Size Reduction," Melbourne Institute Working Paper Series wp2016n37, Melbourne Institute of Applied Economic and Social Research, The University of Melbourne.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:anname:v:589:y:2003:i:1:p:114-149. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.