IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v47y2023i1p11-38.html

Some Lessons From 50 Years of Multiarm Public Policy Experiments

Author

Listed:
  • Larry L. Orr
  • Daniel Gubits

Abstract

In this article, we explore the reasons why multiarm trials have been conducted and the design and analysis issues they involve. We point to three fundamental reasons for such designs: (1) Multiarm designs allow the estimation of “response surfaces†—that is, the variation in response to an intervention across a range of one or more continuous policy parameters. (2) Multiarm designs are an efficient way to test multiple policy approaches to the same social problem simultaneously, either to compare the effects of the different approaches or to estimate the effect of each separately. (3) Multiarm designs may allow for the estimation of the separate and combined effects of discrete program components. We illustrate each of these objectives with examples from the history of public policy experimentation over the past 50 years and discuss some design and analysis issues raised by each, including sample allocation, statistical power, multiple comparisons, and alignment of analysis with goals of the evaluation.

Suggested Citation

  • Larry L. Orr & Daniel Gubits, 2023. "Some Lessons From 50 Years of Multiarm Public Policy Experiments," Evaluation Review, , vol. 47(1), pages 11-38, February.
    • Handle: RePEc:sae:evarev:v:47:y:2023:i:1:p:11-38
    DOI: 10.1177/0193841X20977332
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X20977332
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X20977332?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert A. Moffitt, 1979. "The Labor Supply Response in the Gary Experiment," Journal of Human Resources, University of Wisconsin Press, vol. 14(4), pages 477-487.
    2. Coleman, Stephen, 1996. "The Minnesota income tax compliance experiment: State tax results," MPRA Paper 4827, University Library of Munich, Germany.
    3. Bawden, D. Lee & Harrar, William S., 1977. "Purpose And Design Of The Rural Income Maintenance Experiment," 1977 AAEA-WAEA Joint Meeting, July 31-August 3, San Diego, California 283796, American Agricultural Economics Association (New Name 2008: Agricultural and Applied Economics Association).
    4. Harvey S. Rosen, 1985. "Housing Behavior and the Experimental Housing-Allowance Program: What Have We Learned?," NBER Chapters, in: Social Experimentation, pages 55-94, National Bureau of Economic Research, Inc.
    5. Robert A. Moffitt, 2003. "The Temporary Assistance for Needy Families Program," NBER Chapters, in: Means-Tested Transfer Programs in the United States, pages 291-364, National Bureau of Economic Research, Inc.
    6. Conlisk, John, 1973. "Choice of Response Functional Form in Designing Subsidy Experiments," Econometrica, Econometric Society, vol. 41(4), pages 643-656, July.
    7. Aigner, Dennis J., 1979. "A brief introduction to the methodology of optimal experimental design," Journal of Econometrics, Elsevier, vol. 11(1), pages 7-26, September.
    8. D. Lee Bawden & William S. Harrar, 1977. "Purpose and Design of the Rural Income Maintenance Experiment," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 59(5), pages 855-858.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Larry L. Orr & Daniel Gubits, 2023. "Some Lessons From 50 Years of Multiarm Public Policy Experiments," Evaluation Review, , vol. 47(1), pages 43-70, February.
    2. Connelly, Luke B., 2003. "Balancing the Number and Size of Sites: An Economic Approach to the Optimal Design of Cluster Samples," MPRA Paper 14676, University Library of Munich, Germany.
    3. Saupe, William E., 1980. "Information Needs Relating to Small-Farm Programs and Policies," Economics Statistics and Cooperative Services (ESCS) Reports 329217, United States Department of Agriculture, Economic Research Service.
    4. Madden, Gary & Bloch, Harry & Hensher, David, 1993. "Australian telephone network subscription and calling demands: evidence from a stated-preference experiment," Information Economics and Policy, Elsevier, vol. 5(3), pages 207-230, October.
    5. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    6. James J. Heckman, 1991. "Randomization and Social Policy Evaluation Revisited," NBER Technical Working Papers 0107, National Bureau of Economic Research, Inc.
    7. Nicola Pavoni & G. L. Violante, 2007. "Optimal Welfare-to-Work Programs," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 74(1), pages 283-318.
    8. Dalit Contini & Nicola Negri, 2005. "Would Declining Exit Rates from Welfare Provide Evidence of Welfare Dependence in Homogeneous Environments?," LABORatorio R. Revelli Working Papers Series 39, LABORatorio R. Revelli, Centre for Employment Studies.
    9. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," NBER Working Papers 25072, National Bureau of Economic Research, Inc.
    10. Evenhouse, Eirik & Reilly, Siobhan, 2010. "Multiple-Father Fertility and Arrest Rates," MPRA Paper 22818, University Library of Munich, Germany.
    11. Moffitt, Robert A. & Zahn, Matthew V., 2022. "The Marginal Labor Supply Disincentives of Welfare: Evidence from Administrative Barriers to Participation," Economics Working Paper Archive 66674, The Johns Hopkins University,Department of Economics.
    12. Lemieux, Thomas & Milligan, Kevin, 2008. "Incentive effects of social assistance: A regression discontinuity approach," Journal of Econometrics, Elsevier, vol. 142(2), pages 807-828, February.
    13. Víctor Casero-Alonso & Jesús López-Fidalgo, 2015. "Experimental designs in triangular simultaneous equations models," Statistical Papers, Springer, vol. 56(2), pages 273-290, May.
    14. Christian Holzner & Volker Meier & Martin Werding, 2004. "Time Limits on Welfare Use under Involuntary Unemployment," CESifo Working Paper Series 1220, CESifo.
    15. Maria Cancian & Daniel Meyer, 2014. "Testing the Economic Independence Hypothesis: The Effect of an Exogenous Increase in Child Support on Subsequent Marriage and Cohabitation," Demography, Springer;Population Association of America (PAA), vol. 51(3), pages 857-880, June.
    16. Hallsworth, Michael & List, John A. & Metcalfe, Robert D. & Vlaev, Ivo, 2017. "The behavioralist as tax collector: Using natural field experiments to enhance tax compliance," Journal of Public Economics, Elsevier, vol. 148(C), pages 14-31.
    17. Miguel-Angel Lopez Garcia, 2024. "Subsidios a la vivienda para los hogares de renta baja: un panorama," INVESTIGACIONES REGIONALES - Journal of REGIONAL RESEARCH, Asociación Española de Ciencia Regional, issue 58, pages 33-69.
    18. Laferrere, Anne & Le Blanc, David, 2004. "How do housing allowances affect rents? An empirical analysis of the French case," Journal of Housing Economics, Elsevier, vol. 13(1), pages 36-67, March.
    19. Lentz, Erin C. & Barrett, Christopher B., 2013. "The economics and nutritional impacts of food assistance policies and programs," Food Policy, Elsevier, vol. 42(C), pages 151-163.
    20. Nancy E. Reichman & Hope Corman & Kelly Noonan, 2004. "Effects of Child Health on Sources of Public Support," NBER Working Papers 10762, National Bureau of Economic Research, Inc.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:47:y:2023:i:1:p:11-38. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.