IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v47y2023i1p11-38.html

Some Lessons From 50 Years of Multiarm Public Policy Experiments

Author

Listed:
  • Larry L. Orr
  • Daniel Gubits

Abstract

In this article, we explore the reasons why multiarm trials have been conducted and the design and analysis issues they involve. We point to three fundamental reasons for such designs: (1) Multiarm designs allow the estimation of “response surfaces†—that is, the variation in response to an intervention across a range of one or more continuous policy parameters. (2) Multiarm designs are an efficient way to test multiple policy approaches to the same social problem simultaneously, either to compare the effects of the different approaches or to estimate the effect of each separately. (3) Multiarm designs may allow for the estimation of the separate and combined effects of discrete program components. We illustrate each of these objectives with examples from the history of public policy experimentation over the past 50 years and discuss some design and analysis issues raised by each, including sample allocation, statistical power, multiple comparisons, and alignment of analysis with goals of the evaluation.

Suggested Citation

  • Larry L. Orr & Daniel Gubits, 2023. "Some Lessons From 50 Years of Multiarm Public Policy Experiments," Evaluation Review, , vol. 47(1), pages 11-38, February.
    • Handle: RePEc:sae:evarev:v:47:y:2023:i:1:p:11-38
    DOI: 10.1177/0193841X20977332
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X20977332
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X20977332?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Harvey S. Rosen, 1985. "Housing Behavior and the Experimental Housing-Allowance Program: What Have We Learned?," NBER Chapters, in: Social Experimentation, pages 55-94, National Bureau of Economic Research, Inc.
    2. Robert A. Moffitt, 1979. "The Labor Supply Response in the Gary Experiment," Journal of Human Resources, University of Wisconsin Press, vol. 14(4), pages 477-487.
    3. Robert A. Moffitt, 2003. "The Temporary Assistance for Needy Families Program," NBER Chapters, in: Means-Tested Transfer Programs in the United States, pages 291-364, National Bureau of Economic Research, Inc.
    4. Coleman, Stephen, 1996. "The Minnesota income tax compliance experiment: State tax results," MPRA Paper 4827, University Library of Munich, Germany.
    5. Conlisk, John, 1973. "Choice of Response Functional Form in Designing Subsidy Experiments," Econometrica, Econometric Society, vol. 41(4), pages 643-656, July.
    6. Bawden, D. Lee & Harrar, William S., 1977. "Purpose And Design Of The Rural Income Maintenance Experiment," 1977 AAEA-WAEA Joint Meeting, July 31-August 3, San Diego, California 283796, American Agricultural Economics Association (New Name 2008: Agricultural and Applied Economics Association).
    7. Aigner, Dennis J., 1979. "A brief introduction to the methodology of optimal experimental design," Journal of Econometrics, Elsevier, vol. 11(1), pages 7-26, September.
    8. D. Lee Bawden & William S. Harrar, 1977. "Purpose and Design of the Rural Income Maintenance Experiment," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 59(5), pages 855-858.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Larry L. Orr & Daniel Gubits, 2023. "Some Lessons From 50 Years of Multiarm Public Policy Experiments," Evaluation Review, , vol. 47(1), pages 43-70, February.
    2. Connelly, Luke B., 2003. "Balancing the Number and Size of Sites: An Economic Approach to the Optimal Design of Cluster Samples," MPRA Paper 14676, University Library of Munich, Germany.
    3. Saupe, William E., 1980. "Information Needs Relating to Small-Farm Programs and Policies," Economics Statistics and Cooperative Services (ESCS) Reports 329217, United States Department of Agriculture, Economic Research Service.
    4. Madden, Gary & Bloch, Harry & Hensher, David, 1993. "Australian telephone network subscription and calling demands: evidence from a stated-preference experiment," Information Economics and Policy, Elsevier, vol. 5(3), pages 207-230, October.
    5. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    6. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    7. Nicola Pavoni & G. L. Violante, 2007. "Optimal Welfare-to-Work Programs," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 74(1), pages 283-318.
    8. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    9. Berthold, Norbert & von Berchem, Sascha, 2005. "Lokale Solidarität: die Zukunft der Sozialhilfe?," Discussion Paper Series 76, Julius Maximilian University of Würzburg, Chair of Economic Order and Social Policy.
    10. Peter Lindert, 2003. "Why The Welfare State Looks Like a Free Lunch," Working Papers 59, University of California, Davis, Department of Economics.
    11. Dalit Contini & Nicola Negri, 2005. "Would Declining Exit Rates from Welfare Provide Evidence of Welfare Dependence in Homogeneous Environments?," LABORatorio R. Revelli Working Papers Series 39, LABORatorio R. Revelli, Centre for Employment Studies.
    12. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," NBER Working Papers 25072, National Bureau of Economic Research, Inc.
    13. Evenhouse, Eirik & Reilly, Siobhan, 2010. "Multiple-Father Fertility and Arrest Rates," MPRA Paper 22818, University Library of Munich, Germany.
    14. Moffitt, Robert A. & Zahn, Matthew V., 2022. "The Marginal Labor Supply Disincentives of Welfare: Evidence from Administrative Barriers to Participation," Economics Working Paper Archive 66674, The Johns Hopkins University,Department of Economics.
    15. Ted Joyce & Robert Kaestner & Sanders Korenman & Stanley Henshaw, 2004. "Family Cap Provisions and Changes in Births and Abortions," NBER Working Papers 10214, National Bureau of Economic Research, Inc.
    16. Ayala, Luis & Rodriguez, Magdalena, 2007. "Barriers to employment and welfare dynamics: Evidence from Spain," Journal of Policy Modeling, Elsevier, vol. 29(2), pages 237-257.
    17. Daniel R. Cavagnaro & Richard Gonzalez & Jay I. Myung & Mark A. Pitt, 2013. "Optimal Decision Stimuli for Risky Choice Experiments: An Adaptive Approach," Management Science, INFORMS, vol. 59(2), pages 358-375, February.
    18. Lemieux, Thomas & Milligan, Kevin, 2008. "Incentive effects of social assistance: A regression discontinuity approach," Journal of Econometrics, Elsevier, vol. 142(2), pages 807-828, February.
    19. Víctor Casero-Alonso & Jesús López-Fidalgo, 2015. "Experimental designs in triangular simultaneous equations models," Statistical Papers, Springer, vol. 56(2), pages 273-290, May.
    20. Christian Holzner & Volker Meier & Martin Werding, 2004. "Time Limits on Welfare Use under Involuntary Unemployment," CESifo Working Paper Series 1220, CESifo.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:47:y:2023:i:1:p:11-38. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.