IDEAS home Printed from https://ideas.repec.org/p/cca/wpaper/292.html
   My bibliography  Save this paper

Assignment procedure biases in randomized policy experiments

Author

Listed:
  • Gani Aldashev
  • Georg Kirchsteiger
  • Alexander Sebald

Abstract

Randomized controlled trials (RCT) have become a dominant empirical tool in applied economics. The internal validity of RCTs crucially depends on the (implicit) assumption that the procedure assigning subjects to treatment and control groups has no effect on behavior. We show theoretically that this assumption is violated when people are motivated by belief-dependent preferences and care about the intentions of others. The choice of assignment procedure influences subjects’ behavior and, consequently, the RCTs’ findings. Strikingly, even a credible and explicit randomization procedure does not guarantee an unbiased prediction of the impact of a general introduction of the policy.

Suggested Citation

  • Gani Aldashev & Georg Kirchsteiger & Alexander Sebald, 2012. "Assignment procedure biases in randomized policy experiments," Carlo Alberto Notebooks 292, Collegio Carlo Alberto.
  • Handle: RePEc:cca:wpaper:292
    as

    Download full text from publisher

    File URL: https://www.carloalberto.org/wp-content/uploads/2018/11/no.292.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    2. James J. Heckman, 2010. "Building Bridges between Structural and Program Evaluation Approaches to Evaluating Policy," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 356-398, June.
    3. Esther Duflo & William Gale & Jeffrey Liebman & Peter Orszag & Emmanuel Saez, 2006. "Saving Incentives for Low- and Middle-Income Families: Evidence from a Field Experiment with H&R Block," The Quarterly Journal of Economics, Oxford University Press, vol. 121(4), pages 1311-1346.
    4. Michalopoulos, Charles & Robins, Philip K. & Card, David, 2005. "When financial work incentives pay for themselves: evidence from a randomized social experiment for welfare recipients," Journal of Public Economics, Elsevier, vol. 89(1), pages 5-29, January.
    5. Guido W. Imbens, 2010. "Better LATE Than Nothing: Some Comments on Deaton (2009) and Heckman and Urzua (2009)," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 399-423, June.
    6. Joshua Angrist & Eric Bettinger & Erik Bloom & Elizabeth King & Michael Kremer, 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review, American Economic Association, vol. 92(5), pages 1535-1558, December.
    7. Dufwenberg, Martin & Kirchsteiger, Georg, 2004. "A theory of sequential reciprocity," Games and Economic Behavior, Elsevier, vol. 47(2), pages 268-298, May.
    8. Claudio Ferraz & Frederico Finan, 2008. "Exposing Corrupt Politicians: The Effects of Brazil's Publicly Released Audits on Electoral Outcomes," The Quarterly Journal of Economics, Oxford University Press, vol. 123(2), pages 703-745.
    9. Suresh de Mel & David McKenzie & Christopher Woodruff, 2009. "Returns to Capital in Microenterprises: Evidence from a Field Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 124(1), pages 423-423.
    10. James J. Heckman & Jeffrey A. Smith, 1995. "Assessing the Case for Social Experiments," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 85-110, Spring.
    11. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    12. David Card & Dean R. Hyslop, 2005. "Estimating the Effects of a Time-Limited Earnings Subsidy for Welfare-Leavers," Econometrica, Econometric Society, vol. 73(6), pages 1723-1770, November.
    13. Sebald, Alexander, 2010. "Attribution and reciprocity," Games and Economic Behavior, Elsevier, vol. 68(1), pages 339-352, January.
    14. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    15. Geanakoplos, John & Pearce, David & Stacchetti, Ennio, 1989. "Psychological games and sequential rationality," Games and Economic Behavior, Elsevier, vol. 1(1), pages 60-79, March.
    16. Battigalli, Pierpaolo & Dufwenberg, Martin, 2009. "Dynamic psychological games," Journal of Economic Theory, Elsevier, vol. 144(1), pages 1-35, January.
    17. Paul Schultz, T., 2004. "School subsidies for the poor: evaluating the Mexican Progresa poverty program," Journal of Development Economics, Elsevier, vol. 74(1), pages 199-250, June.
    18. Ruffle, Bradley J., 1999. "Gift giving with emotions," Journal of Economic Behavior & Organization, Elsevier, vol. 39(4), pages 399-420, July.
    19. Joshua Angrist & Eric Bettinger & Michael Kremer, 2006. "Long-Term Educational Consequences of Secondary School Vouchers: Evidence from Administrative Records in Colombia," American Economic Review, American Economic Association, vol. 96(3), pages 847-862, June.
    20. Rabin, Matthew, 1993. "Incorporating Fairness into Game Theory and Economics," American Economic Review, American Economic Association, vol. 83(5), pages 1281-1302, December.
    21. Zephyr, 2010. "The city," City, Taylor & Francis Journals, vol. 14(1-2), pages 154-155, February.
    22. Anup Malani, 2006. "Identifying Placebo Effects with Data from Clinical Trials," Journal of Political Economy, University of Chicago Press, vol. 114(2), pages 236-256, April.
    23. Card, David & Robins, Philip K., 2005. "How important are "entry effects" in financial incentive programs for welfare recipients? Experimental evidence from the Self-Sufficiency Project," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 113-139.
    24. Gary Burtless, 1995. "The Case for Randomized Field Trials in Economic and Policy Research," Journal of Economic Perspectives, American Economic Association, vol. 9(2), pages 63-84, Spring.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Dufwenberg, Martin & Kirchsteiger, Georg, 2019. "Modelling kindness," Journal of Economic Behavior & Organization, Elsevier, vol. 167(C), pages 228-234.
    2. Pierpaolo Battigalli & Martin Dufwenberg, 2020. "Belief-Dependent Motivations and Psychological Game Theory," CESifo Working Paper Series 8285, CESifo.
    3. Elias Bouacida & Renaud Foucart, 2020. "The acceptability of lotteries in allocation problems," Working Papers 301646245, Lancaster University Management School, Economics Department.
    4. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," World Bank Research Observer, World Bank Group, vol. 33(1), pages 34-64.
    5. Gwenolé Le Velly & Céline Dutilly, 2016. "Evaluating Payments for Environmental Services: Methodological Challenges," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-20, February.
    6. Bierbrauer, Felix & Netzer, Nick, 2016. "Mechanism design and intentions," Journal of Economic Theory, Elsevier, vol. 163(C), pages 557-603.
    7. Pierpaolo Battigalli & Martin Dufwenberg, 2019. "Psychological Game Theory," Working Papers 646, IGIER (Innocenzo Gasparini Institute for Economic Research), Bocconi University.
    8. Charles Bellemare & Alexander Sebald, 2019. "Measuring Belief-Dependent Preferences without Information about Beliefs," CESifo Working Paper Series 7505, CESifo.
    9. Kiessling, Lukas & Radbruch, Jonas & Schaube, Sebastian, 2018. "The Impact of Self-Selection on Performance," IZA Discussion Papers 11365, Institute of Labor Economics (IZA).
    10. Omar Al-Ubaydli & John List & Claire Mackevicius & Min Sok Lee & Dana Suskind, 2019. "How Can Experiments Play a Greater Role in Public Policy? 12 Proposals from an Economic Model of Scaling," Artefactual Field Experiments 00679, The Field Experiments Website.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Bierbrauer, Felix & Netzer, Nick, 2016. "Mechanism design and intentions," Journal of Economic Theory, Elsevier, vol. 163(C), pages 557-603.
    3. von Siemens, Ferdinand A., 2013. "Intention-based reciprocity and the hidden costs of control," Journal of Economic Behavior & Organization, Elsevier, vol. 92(C), pages 55-65.
    4. Dufwenberg, Martin & Kirchsteiger, Georg, 2019. "Modelling kindness," Journal of Economic Behavior & Organization, Elsevier, vol. 167(C), pages 228-234.
    5. Aldashev, Gani & Kirchsteiger, Georg & Sebald, Alexander, 2009. "Decision-making Procedures: A General Theory and Its Field Experimental Test," CEPR Discussion Papers 7365, C.E.P.R. Discussion Papers.
    6. Sarkar, Sumit, 2019. "Gratitude, conscience, and reciprocity: Models of supplier motivation when quality is non-contractible," European Journal of Operational Research, Elsevier, vol. 277(2), pages 633-642.
    7. Miettinen, Topi, 2013. "Promises and conventions – An approach to pre-play agreements," Games and Economic Behavior, Elsevier, vol. 80(C), pages 68-84.
    8. Philippe Grégoire, 2018. "Psychology at work," International Review of Economics, Springer;Happiness Economics and Interpersonal Relations (HEIRS), vol. 65(2), pages 119-135, June.
    9. Pikulina, Elena S. & Tergiman, Chloe, 2020. "Preferences for power," Journal of Public Economics, Elsevier, vol. 185(C).
    10. Livio, Luca & De Chiara, Alessandro, 2019. "Friends or foes? Optimal incentives for reciprocal agents," Journal of Economic Behavior & Organization, Elsevier, vol. 167(C), pages 245-278.
    11. Jens Gudmundsson & Jens Leth Hougaard, 2020. "Enabling reciprocity through blockchain design," IFRO Working Paper 2020/14, University of Copenhagen, Department of Food and Resource Economics, revised 09 Feb 2021.
    12. Sebald, Alexander & Vikander, Nick, 2019. "Optimal firm behavior with consumer social image concerns and asymmetric information," Journal of Economic Behavior & Organization, Elsevier, vol. 167(C), pages 311-330.
    13. Battigalli, Pierpaolo & Dufwenberg, Martin, 2009. "Dynamic psychological games," Journal of Economic Theory, Elsevier, vol. 144(1), pages 1-35, January.
    14. Dufwenberg, Martin & Patel, Amrish, 2019. "Introduction to special issue on psychological game theory," Journal of Economic Behavior & Organization, Elsevier, vol. 167(C), pages 181-184.
    15. Sylvain Chassang & Gerard Padro I Miquel & Erik Snowberg, 2012. "Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments," American Economic Review, American Economic Association, vol. 102(4), pages 1279-1309, June.
    16. Jiabin Wu, 2018. "Indirect higher order beliefs and cooperation," Experimental Economics, Springer;Economic Science Association, vol. 21(4), pages 858-876, December.
    17. Andreozzi, Luciano, 2013. "Learning to be fair," Journal of Economic Behavior & Organization, Elsevier, vol. 90(C), pages 181-195.
    18. Gill, David & Stone, Rebecca, 2015. "Desert and inequity aversion in teams," Journal of Public Economics, Elsevier, vol. 123(C), pages 42-54.
    19. Dufwenberg, Martin & Patel, Amrish, 2017. "Reciprocity networks and the participation problem," Games and Economic Behavior, Elsevier, vol. 101(C), pages 260-272.
    20. Baldwin, Kate & Bhavnani, Rikhil R., 2013. "Ancillary Experiments: Opportunities and Challenges," WIDER Working Paper Series 024, World Institute for Development Economic Research (UNU-WIDER).

    More about this item

    Keywords

    Randomized controlled trials; Policy experiments; Internal validity; Procedural concerns; Psychological game theory.;
    All these keywords.

    JEL classification:

    • C70 - Mathematical and Quantitative Methods - - Game Theory and Bargaining Theory - - - General
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D63 - Microeconomics - - Welfare Economics - - - Equity, Justice, Inequality, and Other Normative Criteria and Measurement
    • I38 - Health, Education, and Welfare - - Welfare, Well-Being, and Poverty - - - Government Programs; Provision and Effects of Welfare Programs
    • O12 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development - - - Microeconomic Analyses of Economic Development

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cca:wpaper:292. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/fccaait.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Giovanni Bert (email available below). General contact details of provider: https://edirc.repec.org/data/fccaait.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.