IDEAS home Printed from https://ideas.repec.org/a/taf/jnlbes/v40y2022i4p1552-1568.html
   My bibliography  Save this article

Multiple Testing and the Distributional Effects of Accountability Incentives in Education

Author

Listed:
  • Steven F. Lehrer
  • R. Vincent Pohl
  • Kyungchul Song

Abstract

This article proposes bootstrap-based multiple testing procedures for quantile treatment effect (QTE) heterogeneity under the assumption of selection on observables, and shows its asymptotic validity. Our procedure can be used to detect the quantiles and subgroups exhibiting treatment effect heterogeneity. We apply the multiple testing procedures to data from a large-scale Pakistani school report card experiment, and uncover evidence of policy-relevant heterogeneous effects from information provision on child test scores. Furthermore, our analysis reinforces the importance of preventing the inflation of false positive conclusions because 63% of statistically significant QTEs become insignificant once corrections for multiple testing are applied.

Suggested Citation

  • Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2022. "Multiple Testing and the Distributional Effects of Accountability Incentives in Education," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 40(4), pages 1552-1568, October.
  • Handle: RePEc:taf:jnlbes:v:40:y:2022:i:4:p:1552-1568
    DOI: 10.1080/07350015.2021.1941055
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/07350015.2021.1941055
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1080/07350015.2021.1941055?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jason Brown & Mark Duggan & Ilyana Kuziemko & William Woolston, 2014. "How Does Risk Selection Respond to Risk Adjustment? New Evidence from the Medicare Advantage Program," American Economic Review, American Economic Association, vol. 104(10), pages 3335-3364, October.
    2. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927RR, Cowles Foundation for Research in Economics, Yale University, revised Apr 2015.
    3. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    4. Weili Ding & Steven F. Lehrer, 2007. "Do Peers Affect Student Achievement in China's Secondary Schools?," The Review of Economics and Statistics, MIT Press, vol. 89(2), pages 300-312, May.
    5. Richard K. Crump & V. Joseph Hotz & Guido W. Imbens & Oscar A. Mitnik, 2008. "Nonparametric Tests for Treatment Effect Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 389-405, August.
    6. Sergio Firpo, 2007. "Efficient Semiparametric Estimation of Quantile Treatment Effects," Econometrica, Econometric Society, vol. 75(1), pages 259-276, January.
    7. Alberto Abadie & Joshua Angrist & Guido Imbens, 2002. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Econometrica, Econometric Society, vol. 70(1), pages 91-117, January.
    8. James J. Heckman & Edward Vytlacil, 2005. "Structural Equations, Treatment Effects, and Econometric Policy Evaluation," Econometrica, Econometric Society, vol. 73(3), pages 669-738, May.
    9. Caroline M. Hoxby, 2003. "The Economics of School Choice," NBER Books, National Bureau of Economic Research, Inc, number hox03-1, March.
    10. Pierre Koning & Karen Wiel, 2012. "School Responsiveness to Quality Rankings: An Empirical Analysis of Secondary Education in the Netherlands," De Economist, Springer, vol. 160(4), pages 339-355, December.
    11. James J. Heckman & Jeffrey Smith & Nancy Clements, 1997. "Making The Most Out Of Programme Evaluations and Social Experiments: Accounting For Heterogeneity in Programme Impacts," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 487-535.
    12. Toru Kitagawa & Aleksey Tetenov, 2018. "Who Should Be Treated? Empirical Welfare Maximization Methods for Treatment Choice," Econometrica, Econometric Society, vol. 86(2), pages 591-616, March.
    13. Philip Oreopoulos & Daniel Lang & Joshua Angrist, 2009. "Incentives and Services for College Achievement: Evidence from a Randomized Trial," American Economic Journal: Applied Economics, American Economic Association, vol. 1(1), pages 136-163, January.
    14. Milgrom, Paul & Roberts, John, 1986. "Price and Advertising Signals of Product Quality," Journal of Political Economy, University of Chicago Press, vol. 94(4), pages 796-821, August.
    15. Camargo, Braz & Camelo, Rafael & Firpo, Sergio & Ponczek, Vladimir, 2014. "Information, Market Incentives, and Student Performance," IZA Discussion Papers 7941, Institute of Labor Economics (IZA).
    16. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja, 2017. "Report Cards: The Impact of Providing School and Child Test Scores on Educational Markets," American Economic Review, American Economic Association, vol. 107(6), pages 1535-1563, June.
    17. Heckman, James J. & Urzúa, Sergio, 2010. "Comparing IV with structural models: What simple IV can and cannot identify," Journal of Econometrics, Elsevier, vol. 156(1), pages 27-37, May.
    18. Jane Friesen & Mohsen Javdani & Justin Smith & Simon Woodcock, 2012. "How do school ‘report cards’ affect school choice decisions?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 45(2), pages 784-807, May.
    19. Hoxby, Caroline M. (ed.), 2003. "The Economics of School Choice," National Bureau of Economic Research Books, University of Chicago Press, number 9780226355337.
    20. Robert W. Fairlie & Jonathan Robinson, 2013. "Experimental Evidence on the Effects of Home Computers on Academic Achievement among Schoolchildren," American Economic Journal: Applied Economics, American Economic Association, vol. 5(3), pages 211-240, July.
    21. Caroline Minter Hoxby, 2003. "School Choice and School Productivity. Could School Choice Be a Tide that Lifts All Boats?," NBER Chapters, in: The Economics of School Choice, pages 287-342, National Bureau of Economic Research, Inc.
    22. G�nther Fink & Margaret McConnell & Sebastian Vollmer, 2014. "Testing for heterogeneous treatment effects in experimental data: false discovery risks and correction procedures," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(1), pages 44-57, January.
    23. Luc Behaghel & Clément de Chaisemartin & Marc Gurgand, 2017. "Ready for Boarding? The Effects of a Boarding School for Disadvantaged Students," American Economic Journal: Applied Economics, American Economic Association, vol. 9(1), pages 140-164, January.
    24. Joseph P. Romano & Azeem M. Shaikh, 2010. "Inference for the Identified Set in Partially Identified Econometric Models," Econometrica, Econometric Society, vol. 78(1), pages 169-211, January.
    25. Jason Allen & Robert Clark & Jean-Fran?ois Houde, 2014. "The Effect of Mergers in Search Markets: Evidence from the Canadian Mortgage Industry," American Economic Review, American Economic Association, vol. 104(10), pages 3365-3396, October.
    26. James J. Heckman, 2001. "Micro Data, Heterogeneity, and the Evaluation of Public Policy: Nobel Lecture," Journal of Political Economy, University of Chicago Press, vol. 109(4), pages 673-748, August.
    27. Bruno Crépon & Florencia Devoto & Esther Duflo & William Parienté, 2015. "Estimating the Impact of Microcredit on Those Who Take It Up: Evidence from a Randomized Experiment in Morocco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 123-150, January.
    28. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    29. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    30. Weili Ding & Steven F. Lehrer, 2007. "Do Peers Affect Student Achievement in China's Secondary Schools?," The Review of Economics and Statistics, MIT Press, vol. 89(2), pages 300-312, May.
    31. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    32. Asher Wolinsky, 1983. "Prices as Signals of Product Quality," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 50(4), pages 647-658.
    33. Carl Shapiro, 1983. "Premiums for High Quality Products as Returns to Reputations," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 98(4), pages 659-679.
    34. Rothe, Christoph, 2010. "Nonparametric estimation of distributional policy effects," Journal of Econometrics, Elsevier, vol. 155(1), pages 56-70, March.
    35. Bengt Holmstrom, 1999. "Managerial Incentive Problems: A Dynamic Perspective," NBER Working Papers 6875, National Bureau of Economic Research, Inc.
    36. Mark Schneider & Gregory Elacqua & Jack Buckley, 2006. "School choice in Chile: Is it class or the classroom?," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 25(3), pages 577-601.
    37. Maier, Michael, 2011. "Tests for distributional treatment effects under unconfoundedness," Economics Letters, Elsevier, vol. 110(1), pages 49-51, January.
    38. Christian N. Brinch & Magne Mogstad & Matthew Wiswall, 2017. "Beyond LATE with a Discrete Instrument," Journal of Political Economy, University of Chicago Press, vol. 125(4), pages 985-1039.
    39. Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
    40. Charles F. Manski, 2004. "Statistical Treatment Rules for Heterogeneous Populations," Econometrica, Econometric Society, vol. 72(4), pages 1221-1246, July.
    41. Justine S. Hastings & Jeffrey M. Weinstein, 2008. "Information, School Choice, and Academic Achievement: Evidence from Two Experiments," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(4), pages 1373-1414.
    42. Mizala, Alejandra & Urquiola, Miguel, 2013. "School markets: The impact of information approximating schools' effectiveness," Journal of Development Economics, Elsevier, vol. 103(C), pages 313-335.
    43. Caroline Minter Hoxby, 2003. "Introduction to "The Economics of School Choice"," NBER Chapters, in: The Economics of School Choice, pages 1-22, National Bureau of Economic Research, Inc.
    44. Dehejia, Rajeev H., 2005. "Program evaluation as a decision problem," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 141-173.
    45. Bengt Holmström, 1999. "Managerial Incentive Problems: A Dynamic Perspective," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 66(1), pages 169-182.
    46. Abadie A., 2002. "Bootstrap Tests for Distributional Treatment Effects in Instrumental Variable Models," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 284-292, March.
    47. Stephen Gibbons & Stephen Machin, 2006. "Paying for Primary Schools: Admission Constraints, School Popularity or Congestion?," Economic Journal, Royal Economic Society, vol. 116(510), pages 77-92, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Brennan S Thompson & Matthew D Webb, 2019. "A simple, graphical approach to comparing multiple treatments," The Econometrics Journal, Royal Economic Society, vol. 22(2), pages 188-205.
    2. Zongwu Cai & Ying Fang & Ming Lin & Shengfang Tang, 2021. "A Nonparametric Test for Testing Heterogeneity in Conditional Quantile Treatment Effects," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202117, University of Kansas, Department of Economics, revised Aug 2021.
    3. Zongwu Cai & Ying Fang & Ming Lin & Shengfang Tang, 2020. "Inferences for Partially Conditional Quantile Treatment Effect Model," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202005, University of Kansas, Department of Economics, revised Feb 2020.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
    2. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    3. Victor Chernozhukov & Iván Fernández‐Val & Blaise Melly, 2013. "Inference on Counterfactual Distributions," Econometrica, Econometric Society, vol. 81(6), pages 2205-2268, November.
    4. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2017. "Can Variation in Subgroups' Average Treatment Effects Explain Treatment Effect Heterogeneity? Evidence from a Social Experiment," The Review of Economics and Statistics, MIT Press, vol. 99(4), pages 683-697, July.
    5. Sokbae Lee & Yoon-Jae Whang, 2009. "Nonparametric Tests of Conditional Treatment Effects," Cowles Foundation Discussion Papers 1740, Cowles Foundation for Research in Economics, Yale University.
    6. Blaise Melly und Kaspar W thrich, 2016. "Local quantile treatment effects," Diskussionsschriften dp1605, Universitaet Bern, Departement Volkswirtschaft.
    7. Słoczyński, Tymon, 2012. "New Evidence on Linear Regression and Treatment Effect Heterogeneity," MPRA Paper 39524, University Library of Munich, Germany.
    8. Sakos, Grayson & Cerulli, Giovanni & Garbero, Alessandra, 2021. "Beyond the ATE: Idiosyncratic Effect Estimation to Uncover Distributional Impacts Results from 17 Impact Evaluations," 2021 Annual Meeting, August 1-3, Austin, Texas 314017, Agricultural and Applied Economics Association.
    9. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927RR, Cowles Foundation for Research in Economics, Yale University, revised Apr 2015.
    10. Augustine Denteh & Helge Liebert, 2022. "Who Increases Emergency Department Use? New Insights from the Oregon Health Insurance Experiment," Working Papers 2201, Tulane University, Department of Economics.
    11. Huber, Martin, 2019. "An introduction to flexible methods for policy evaluation," FSES Working Papers 504, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    12. Black, Dan A. & Joo, Joonhwi & LaLonde, Robert & Smith, Jeffrey A. & Taylor, Evan J., 2022. "Simple Tests for Selection: Learning More from Instrumental Variables," Labour Economics, Elsevier, vol. 79(C).
    13. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2006. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," American Economic Review, American Economic Association, vol. 96(4), pages 988-1012, September.
    14. Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl & Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl, 2017. "Distributional impact analysis: toolkit and illustrations of impacts beyond the average treatment effect," Policy Research Working Paper Series 8139, The World Bank.
    15. Susan Athey & Guido Imbens, 2016. "The Econometrics of Randomized Experiments," Papers 1607.00698, arXiv.org.
    16. Huber, Martin & Wüthrich, Kaspar, 2017. "Evaluating local average and quantile treatment effects under endogeneity based on instruments: a review," FSES Working Papers 479, Faculty of Economics and Social Sciences, University of Freiburg/Fribourg Switzerland.
    17. Machado, Cecilia & Szerman, Christiane, 2016. "Centralized Admission and the Student-College Match," IZA Discussion Papers 10251, Institute of Labor Economics (IZA).
    18. W. Bentley MacLeod & Miguel Urquiola, 2009. "Anti-Lemons: School Reputation and Educational Quality," NBER Working Papers 15112, National Bureau of Economic Research, Inc.
    19. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja, 2017. "Report Cards: The Impact of Providing School and Child Test Scores on Educational Markets," American Economic Review, American Economic Association, vol. 107(6), pages 1535-1563, June.
    20. Firpo, Sergio & Galvao, Antonio F. & Kobus, Martyna & Parker, Thomas & Rosa-Dias, Pedro, 2020. "Loss Aversion and the Welfare Ranking of Policy Interventions," IZA Discussion Papers 13176, Institute of Labor Economics (IZA).

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • L15 - Industrial Organization - - Market Structure, Firm Strategy, and Market Performance - - - Information and Product Quality

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:jnlbes:v:40:y:2022:i:4:p:1552-1568. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/UBES20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.