IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v39y2014i6p478-501.html
   My bibliography  Save this article

How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations

Author

Listed:
  • Elizabeth Tipton

Abstract

Although a large-scale experiment can provide an estimate of the average causal impact for a program, the sample of sites included in the experiment is often not drawn randomly from the inference population of interest. In this article, we provide a generalizability index that can be used to assess the degree of similarity between the sample of units in an experiment and one or more inference populations on a set of selected covariates. The index takes values between 0 and 1 and indicates both when a sample is like a miniature of the population and how well reweighting methods may perform when differences exist. Results of simulation studies are provided that develop rules of thumb for interpretation as well as an example.

Suggested Citation

  • Elizabeth Tipton, 2014. "How Generalizable Is Your Experiment? An Index for Comparing Experimental Samples and Populations," Journal of Educational and Behavioral Statistics, , vol. 39(6), pages 478-501, December.
    • Handle: RePEc:sae:jedbes:v:39:y:2014:i:6:p:478-501
    DOI: 10.3102/1076998614558486
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998614558486
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998614558486?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert B. Olsen & Larry L. Orr & Stephen H. Bell & Elizabeth A. Stuart, 2013. "External Validity in Policy Evaluations That Choose Sites Purposively," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(1), pages 107-121, January.
    2. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    3. Evelyn Kitagawa, 1964. "Standardized comparisons in population research," Demography, Springer;Population Association of America (PAA), vol. 1(1), pages 296-315, March.
    4. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Wendy Chan, 2018. "Applications of Small Area Estimation to Generalization With Subclassification by Propensity Scores," Journal of Educational and Behavioral Statistics, , vol. 43(2), pages 182-224, April.
    2. Sharples, Linda D., 2018. "The role of statistics in the era of big data: Electronic health records for healthcare research," Statistics & Probability Letters, Elsevier, vol. 136(C), pages 105-110.
    3. Elizabeth A. Stuart & Anna Rhodes, 2017. "Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding Sufficient Data," Evaluation Review, , vol. 41(4), pages 357-388, August.
    4. Elizabeth Tipton, 2021. "Beyond generalization of the ATE: Designing randomized trials to understand treatment effect heterogeneity," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 504-521, April.
    5. Elizabeth Tipton & Kelly Hallberg & Larry V. Hedges & Wendy Chan, 2017. "Implications of Small Samples for Generalization: Adjustments and Rules of Thumb," Evaluation Review, , vol. 41(5), pages 472-505, October.
    6. Bo Zhang, 2023. "Efficient algorithms for building representative matched pairs with enhanced generalizability," Biometrics, The International Biometric Society, vol. 79(4), pages 3981-3997, December.
    7. Sarah A. Avellar & Jaime Thomas & Rebecca Kleinman & Emily Sama-Miller & Sara E. Woodruff & Rebecca Coughlin & T’Pring R. Westbrook, 2017. "External Validity: The Next Step for Systematic Reviews?," Evaluation Review, , vol. 41(4), pages 283-325, August.
    8. Esterling, Kevin & Brady, David & Schwitzgebel, Eric, 2021. "The Necessity of Construct and External Validity for Generalized Causal Claims," OSF Preprints 2s8w5, Center for Open Science.
    9. Elizabeth Tipton, 2013. "Improving Generalizations From Experiments Using Propensity Score Subclassification," Journal of Educational and Behavioral Statistics, , vol. 38(3), pages 239-266, June.
    10. Rajeev Dehejia & Cristian Pop-Eleches & Cyrus Samii, 2021. "From Local to Global: External Validity in a Fertility Natural Experiment," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(1), pages 217-243, January.
    11. repec:mpr:mprres:8128 is not listed on IDEAS
    12. Andrews, Isaiah & Oster, Emily, 2019. "A simple approximation for evaluating external validity bias," Economics Letters, Elsevier, vol. 178(C), pages 58-62.
    13. Elizabeth Tipton & Laura R. Peck, 2017. "A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations," Evaluation Review, , vol. 41(4), pages 326-356, August.
    14. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.
    15. Jacob Alex Klerman, 2017. "Editor in Chief’s Comment: External Validity in Systematic Reviews," Evaluation Review, , vol. 41(5), pages 391-402, October.
    16. Xinkun Nie & Guido Imbens & Stefan Wager, 2021. "Covariate Balancing Sensitivity Analysis for Extrapolating Randomized Trials across Locations," Papers 2112.04723, arXiv.org.
    17. Liu, Yan & Chen, Xi & Yan, Zhijun, 2019. "Depression in the House: The Effects of Household Air Pollution from Solid Fuel Use in China," IZA Discussion Papers 12654, Institute of Labor Economics (IZA).
    18. Noemi Kreif & Richard Grieve & Rosalba Radice & Zia Sadique & Roland Ramsahai & Jasjeet S. Sekhon, 2012. "Methods for Estimating Subgroup Effects in Cost-Effectiveness Analyses That Use Observational Data," Medical Decision Making, , vol. 32(6), pages 750-763, November.
    19. Do, Manh Hung & Nguyen, Trung Thanh & Grote, Ulrike, 2023. "Land consolidation, rice production, and agricultural transformation: Evidence from household panel data for Vietnam," Economic Analysis and Policy, Elsevier, vol. 77(C), pages 157-173.
    20. Federico Biagi & Daniele Bondonio & Alberto Martini, 2015. "Counterfactual Impact Evaluation of Enterprise Support Programmes. Evidence from a Decade of Subsidies to Italian Firm," ERSA conference papers ersa15p1619, European Regional Science Association.
    21. Bono, Pierre-Henri & David, Quentin & Desbordes, Rodolphe & Py, Loriane, 2022. "Metro infrastructure and metropolitan attractiveness," Regional Science and Urban Economics, Elsevier, vol. 93(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:39:y:2014:i:6:p:478-501. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.