IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i4p357-388.html
   My bibliography  Save this article

Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding Sufficient Data

Author

Listed:
  • Elizabeth A. Stuart
  • Anna Rhodes

Abstract

Background: Given increasing concerns about the relevance of research to policy and practice, there is growing interest in assessing and enhancing the external validity of randomized trials: determining how useful a given randomized trial is for informing a policy question for a specific target population. Objectives: This article highlights recent advances in assessing and enhancing external validity, with a focus on the data needed to make ex post statistical adjustments to enhance the applicability of experimental findings to populations potentially different from their study sample. Research design: We use a case study to illustrate how to generalize treatment effect estimates from a randomized trial sample to a target population, in particular comparing the sample of children in a randomized trial of a supplemental program for Head Start centers (the Research-Based, Developmentally Informed study) to the national population of children eligible for Head Start, as represented in the Head Start Impact Study. Results: For this case study, common data elements between the trial sample and population were limited, making reliable generalization from the trial sample to the population challenging. Conclusions: To answer important questions about external validity, more publicly available data are needed. In addition, future studies should make an effort to collect measures similar to those in other data sets. Measure comparability between population data sets and randomized trials that use samples of convenience will greatly enhance the range of research and policy relevant questions that can be answered.

Suggested Citation

  • Elizabeth A. Stuart & Anna Rhodes, 2017. "Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding Sufficient Data," Evaluation Review, , vol. 41(4), pages 357-388, August.
  • Handle: RePEc:sae:evarev:v:41:y:2017:i:4:p:357-388
    DOI: 10.1177/0193841X16660663
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X16660663
    Download Restriction: no

    File URL: https://libkey.io/10.1177/0193841X16660663?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert B. Olsen & Larry L. Orr & Stephen H. Bell & Elizabeth A. Stuart, 2013. "External Validity in Policy Evaluations That Choose Sites Purposively," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(1), pages 107-121, January.
    2. Thomas D. Cook, 2014. "Generalizing Causal Knowledge In The Policy Sciences: External Validity As A Task Of Both Multiattribute Representation And Multiattribute Extrapolation," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(2), pages 527-536, March.
    3. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    4. Anonymous, 2013. "Introduction to the Issue," Journal of Wine Economics, Cambridge University Press, vol. 8(2), pages 129-130, November.
    5. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    6. Colm O'Muircheartaigh & Larry V. Hedges, 2014. "Generalizing from unrepresentative experiments: a stratified propensity score approach," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 63(2), pages 195-210, February.
    7. Anonymous, 2013. "Introduction to the Issue," Journal of Wine Economics, Cambridge University Press, vol. 8(3), pages 243-243, December.
    8. repec:mpr:mprres:8128 is not listed on IDEAS
    9. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. John M. Brooks & Cole G. Chapman & Mary C. Schroeder, 2018. "Understanding Treatment Effect Estimates When Treatment Effects Are Heterogeneous for More Than One Outcome," Applied Health Economics and Health Policy, Springer, vol. 16(3), pages 381-393, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:mpr:mprres:8128 is not listed on IDEAS
    2. Wendy Chan, 2018. "Applications of Small Area Estimation to Generalization With Subclassification by Propensity Scores," Journal of Educational and Behavioral Statistics, , vol. 43(2), pages 182-224, April.
    3. Elizabeth Tipton & Kelly Hallberg & Larry V. Hedges & Wendy Chan, 2017. "Implications of Small Samples for Generalization: Adjustments and Rules of Thumb," Evaluation Review, , vol. 41(5), pages 472-505, October.
    4. Elizabeth Tipton & Laura R. Peck, 2017. "A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations," Evaluation Review, , vol. 41(4), pages 326-356, August.
    5. Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    6. Jacob Alex Klerman, 2017. "Editor in Chief’s Comment: External Validity in Systematic Reviews," Evaluation Review, , vol. 41(5), pages 391-402, October.
    7. Esterling, Kevin & Brady, David & Schwitzgebel, Eric, 2021. "The Necessity of Construct and External Validity for Generalized Causal Claims," OSF Preprints 2s8w5, Center for Open Science.
    8. Ranjana Raghunathan, 2022. "Everyday Intimacies and Inter-Ethnic Relationships: Tracing Entanglements of Gender and Race in Multicultural Singapore," Sociological Research Online, , vol. 27(1), pages 77-94, March.
    9. Songsore, Emmanuel & Buzzelli, Michael, 2014. "Social responses to wind energy development in Ontario: The influence of health risk perceptions and associated concerns," Energy Policy, Elsevier, vol. 69(C), pages 285-296.
    10. Tapsuwan, Sorada & Polyakov, Maksym & Bark, Rosalind & Nolan, Martin, 2015. "Valuing the Barmah–Millewa Forest and in stream river flows: A spatial heteroskedasticity and autocorrelation consistent (SHAC) approach," Ecological Economics, Elsevier, vol. 110(C), pages 98-105.
    11. Bertschek, Irene & Kesler, Reinhold, 2022. "Let the user speak: Is feedback on Facebook a source of firms’ innovation?," Information Economics and Policy, Elsevier, vol. 60(C).
    12. Gigi Foster, 2020. "The behavioural economics of government responses to COVID-19," Journal of Behavioral Economics for Policy, Society for the Advancement of Behavioral Economics (SABE), vol. 4(S3), pages 11-43, December.
    13. Balint, T. & Lamperti, F. & Mandel, A. & Napoletano, M. & Roventini, A. & Sapio, A., 2017. "Complexity and the Economics of Climate Change: A Survey and a Look Forward," Ecological Economics, Elsevier, vol. 138(C), pages 252-265.
    14. Audoly, Richard & Vogt-Schilb, Adrien & Guivarch, Céline & Pfeiffer, Alexander, 2018. "Pathways toward zero-carbon electricity required for climate stabilization," Applied Energy, Elsevier, vol. 225(C), pages 884-901.
    15. Gerards, Ruud & Welters, Ricardo, 2016. "Impact of financial pressure on unemployed job search, job find success and job quality," ROA Research Memorandum 008, Maastricht University, Research Centre for Education and the Labour Market (ROA).
    16. Vasile-Daniel Păvăloaia & Elena-Mădălina Teodor & Doina Fotache & Magdalena Danileţ, 2019. "Opinion Mining on Social Media Data: Sentiment Analysis of User Preferences," Sustainability, MDPI, vol. 11(16), pages 1-21, August.
    17. Cailong Xu & Ruidong Li & Wenwen Song & Tingting Wu & Shi Sun & Shuixiu Hu & Tianfu Han & Cunxiang Wu, 2021. "Responses of Branch Number and Yield Component of Soybean Cultivars Tested in Different Planting Densities," Agriculture, MDPI, vol. 11(1), pages 1-12, January.
    18. Caitlin Robinson & Stefan Bouzarovski & Sarah Lindley, 2018. "Underrepresenting neighbourhood vulnerabilities? The measurement of fuel poverty in England," Environment and Planning A, , vol. 50(5), pages 1109-1127, August.
    19. Michaela Haase & Emmanuel Raufflet, 2017. "Ideologies in Markets, Organizations, and Business Ethics: Drafting a Map: Introduction to the Special Issue," Journal of Business Ethics, Springer, vol. 142(4), pages 629-639, June.
    20. Rafael Alcadipani & Cíntia Rodrigues Oliveira Medeiros, 2020. "When Corporations Cause Harm: A Critical View of Corporate Social Irresponsibility and Corporate Crimes," Journal of Business Ethics, Springer, vol. 167(2), pages 285-297, November.
    21. Mansoora Ahmed & Sun Zehou & Syed Ali Raza & Muhammad Asif Qureshi & Sara Qamar Yousufi, 2020. "Impact of CSR and environmental triggers on employee green behavior: The mediating effect of employee well‐being," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 27(5), pages 2225-2239, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:4:p:357-388. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.