IDEAS home Printed from https://ideas.repec.org/p/cpr/ceprdp/11361.html
   My bibliography  Save this paper

The Paradox of Policy-Relevant RCTs and Natural Experiments

Author

Listed:
  • Chemla, Gilles
  • Hennessy, Christopher

Abstract

According to conventional wisdom, RCTs and natural experiments represent especially credible bases for econometric inference, facilitating evidence-based policymaking. We assess credibility in dynamic settings, examining robustness of evidence derived from an exogenous first-stage randomization applied to measure zero subjects. If government is able (unable) to alter policy in response, experimental evidence is contaminated (uncontaminated) by ex post policy endogeneity: Measured responses depend upon the government objective function into which the evidence will be fed. Similarly, if government perceives experimental evidence as credible (non-credible), the very act of observation changes (does not change) agent behavior. Thus, paradoxically, the experimental evidence is contaminated if and only if government is willing and able to use it. Ex post endogeneity causes measured responses to hinge upon (unknown) parameters of the governmental objective function, as well as a priori beliefs regarding the causal parameters to be estimated. Moreover, heterogeneous causal effect parameters induce endogenous belief heterogeneity. This link between beliefs and causal effect parameters makes it difficult, and potentially impossible, to isolate the latter using experimental evidence. Finally, we show treatment-control differences in RCTs are contaminated unless stock variable accumulation cost functions satisfy strong functional form assumptions: zero fixed costs, equality of buy and sell prices, and quadratic adjustment costs.

Suggested Citation

  • Chemla, Gilles & Hennessy, Christopher, 2016. "The Paradox of Policy-Relevant RCTs and Natural Experiments," CEPR Discussion Papers 11361, C.E.P.R. Discussion Papers.
  • Handle: RePEc:cpr:ceprdp:11361
    as

    Download full text from publisher

    File URL: http://www.cepr.org/active/publications/discussion_papers/dp.php?dpno=11361
    Download Restriction: CEPR Discussion Papers are free to download for our researchers, subscribers and members. If you fall into one of these categories but have trouble downloading our papers, please contact us at subscribers@cepr.org

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Steven D. Levitt & John A. List, 2011. "Was There Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 3(1), pages 224-238, January.
    3. Aviva Aron-Dine & Liran Einav & Amy Finkelstein, 2013. "The RAND Health Insurance Experiment, Three Decades Later," Journal of Economic Perspectives, American Economic Association, vol. 27(1), pages 197-222, Winter.
    4. Berman, Eli & Bui, Linda T. M., 2001. "Environmental regulation and labor demand: evidence from the South Coast Air Basin," Journal of Public Economics, Elsevier, vol. 79(2), pages 265-295, February.
    5. Sargent, Thomas J, 1971. "A Note on the 'Accelerationist' Controversy," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 3(3), pages 721-725, August.
    6. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    7. Taylor, John B, 1979. "Estimation and Control of a Macroeconomic Model with Rational Expectations," Econometrica, Econometric Society, vol. 47(5), pages 1267-1286, September.
    8. Salemi, Michael K & Sargent, Thomas J, 1979. "The Demand for Money during Hyperinflation under Rational Expectations: II," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 20(3), pages 741-758, October.
    9. W. Reed Walker, 2011. "Environmental Regulation and Labor Reallocation: Evidence from the Clean Air Act," American Economic Review, American Economic Association, vol. 101(3), pages 442-447, May.
    10. Sargent, Thomas J & Wallace, Neil, 1973. "Rational Expectations and the Dynamics of Hyperinflation," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 14(2), pages 328-350, June.
    11. Daron Acemoglu, 2010. "Theory, General Equilibrium, and Political Economy in Development Economics," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 17-32, Summer.
    12. Adnan Q. Khan & Asim I. Khwaja & Benjamin A. Olken, 2016. "Tax Farming Redux: Experimental Evidence on Performance Pay for Tax Collectors," The Quarterly Journal of Economics, Oxford University Press, vol. 131(1), pages 219-271.
    13. Sargent, Thomas J, 1984. "Autoregressions, Expectations, and Advice," American Economic Review, American Economic Association, vol. 74(2), pages 408-415, May.
    14. Kahn, Matthew E. & Mansur, Erin T., 2013. "Do local energy prices and regulation affect the geographic concentration of employment?," Journal of Public Economics, Elsevier, vol. 101(C), pages 105-114.
    15. James J. Heckman, 2000. "Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, Oxford University Press, vol. 115(1), pages 45-97.
    16. Christopher A. Sims, 1982. "Policy Analysis with Econometric Models," Brookings Papers on Economic Activity, Economic Studies Program, The Brookings Institution, vol. 13(1), pages 107-164.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wedow, Michael & Koetter, Michael & Podlich, Natalia, 2017. "Inside asset purchase programs: the effects of unconventional policy on banking competition," Working Paper Series 2017, European Central Bank.

    More about this item

    Keywords

    firms; government; investment; natural policy experiments; pollution; randomized controlled trials;

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cpr:ceprdp:11361. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.