IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Log in (now much improved!) to save this paper

The Paradox of Policy-Relevant RCTs and Natural Experiments

Listed author(s):
  • Chemla, Gilles
  • Hennessy, Christopher
Registered author(s):

    According to conventional wisdom, RCTs and natural experiments represent especially credible bases for econometric inference, facilitating evidence-based policymaking. We assess credibility in dynamic settings, examining robustness of evidence derived from an exogenous first-stage randomization applied to measure zero subjects. If government is able (unable) to alter policy in response, experimental evidence is contaminated (uncontaminated) by ex post policy endogeneity: Measured responses depend upon the government objective function into which the evidence will be fed. Similarly, if government perceives experimental evidence as credible (non-credible), the very act of observation changes (does not change) agent behavior. Thus, paradoxically, the experimental evidence is contaminated if and only if government is willing and able to use it. Ex post endogeneity causes measured responses to hinge upon (unknown) parameters of the governmental objective function, as well as a priori beliefs regarding the causal parameters to be estimated. Moreover, heterogeneous causal effect parameters induce endogenous belief heterogeneity. This link between beliefs and causal effect parameters makes it difficult, and potentially impossible, to isolate the latter using experimental evidence. Finally, we show treatment-control differences in RCTs are contaminated unless stock variable accumulation cost functions satisfy strong functional form assumptions: zero fixed costs, equality of buy and sell prices, and quadratic adjustment costs.

    If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

    File URL: http://www.cepr.org/active/publications/discussion_papers/dp.php?dpno=11361
    Download Restriction: CEPR Discussion Papers are free to download for our researchers, subscribers and members. If you fall into one of these categories but have trouble downloading our papers, please contact us at subscribers@cepr.org

    As the access to this document is restricted, you may want to look for a different version under "Related research" (further below) or search for a different version of it.

    Paper provided by C.E.P.R. Discussion Papers in its series CEPR Discussion Papers with number 11361.

    as
    in new window

    Length:
    Date of creation: Jun 2016
    Handle: RePEc:cpr:ceprdp:11361
    Contact details of provider: Postal:
    Centre for Economic Policy Research, 77 Bastwick Street, London EC1V 3PZ.

    Phone: 44 - 20 - 7183 8801
    Fax: 44 - 20 - 7183 8820

    Order Information: Email:


    References listed on IDEAS
    Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

    as
    in new window


    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Steven D. Levitt & John A. List, 2011. "Was There Really a Hawthorne Effect at the Hawthorne Plant? An Analysis of the Original Illumination Experiments," American Economic Journal: Applied Economics, American Economic Association, vol. 3(1), pages 224-238, January.
    3. Aviva Aron-Dine & Liran Einav & Amy Finkelstein, 2013. "The RAND Health Insurance Experiment, Three Decades Later," Journal of Economic Perspectives, American Economic Association, vol. 27(1), pages 197-222, Winter.
    4. Berman, Eli & Bui, Linda T. M., 2001. "Environmental regulation and labor demand: evidence from the South Coast Air Basin," Journal of Public Economics, Elsevier, vol. 79(2), pages 265-295, February.
    5. Sargent, Thomas J, 1971. "A Note on the 'Accelerationist' Controversy," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 3(3), pages 721-725, August.
    6. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    7. Taylor, John B, 1979. "Estimation and Control of a Macroeconomic Model with Rational Expectations," Econometrica, Econometric Society, vol. 47(5), pages 1267-1286, September.
    8. Salemi, Michael K & Sargent, Thomas J, 1979. "The Demand for Money during Hyperinflation under Rational Expectations: II," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 20(3), pages 741-758, October.
    9. W. Reed Walker, 2011. "Environmental Regulation and Labor Reallocation: Evidence from the Clean Air Act," American Economic Review, American Economic Association, vol. 101(3), pages 442-447, May.
    10. Sargent, Thomas J & Wallace, Neil, 1973. "Rational Expectations and the Dynamics of Hyperinflation," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 14(2), pages 328-350, June.
    11. Daron Acemoglu, 2010. "Theory, General Equilibrium, and Political Economy in Development Economics," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 17-32, Summer.
    12. Adnan Q. Khan & Asim I. Khwaja & Benjamin A. Olken, 2016. "Tax Farming Redux: Experimental Evidence on Performance Pay for Tax Collectors," The Quarterly Journal of Economics, Oxford University Press, vol. 131(1), pages 219-271.
    13. Sargent, Thomas J, 1984. "Autoregressions, Expectations, and Advice," American Economic Review, American Economic Association, vol. 74(2), pages 408-415, May.
    14. Kahn, Matthew E. & Mansur, Erin T., 2013. "Do local energy prices and regulation affect the geographic concentration of employment?," Journal of Public Economics, Elsevier, vol. 101(C), pages 105-114.
    15. James J. Heckman, 2000. "Causal Parameters and Policy Analysis in Economics: A Twentieth Century Retrospective," The Quarterly Journal of Economics, Oxford University Press, vol. 115(1), pages 45-97.
    16. Christopher A. Sims, 1982. "Policy Analysis with Econometric Models," Brookings Papers on Economic Activity, Economic Studies Program, The Brookings Institution, vol. 13(1), pages 107-164.
    Full references (including those not matched with items on IDEAS)

    This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

    When requesting a correction, please mention this item's handle: RePEc:cpr:ceprdp:11361. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ()

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If references are entirely missing, you can add them using this form.

    If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.