IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v38y2013i3p239-266.html
   My bibliography  Save this article

Improving Generalizations From Experiments Using Propensity Score Subclassification

Author

Listed:
  • Elizabeth Tipton

Abstract

As a result of the use of random assignment to treatment, randomized experiments typically have high internal validity. However, units are very rarely randomly selected from a well-defined population of interest into an experiment; this results in low external validity. Under nonrandom sampling, this means that the estimate of the sample average treatment effect calculated in the experiment can be a biased estimate of the population average treatment effect. This article explores the use of the propensity score subclassification estimator as a means for improving generalizations from experiments. It first lays out the assumptions necessary for generalizations, then investigates the amount of bias reduction and average variance inflation that is likely when compared to a conventional estimator. It concludes with a discussion of issues that arise when the population of interest is not well represented by the experiment, and an example.

Suggested Citation

  • Elizabeth Tipton, 2013. "Improving Generalizations From Experiments Using Propensity Score Subclassification," Journal of Educational and Behavioral Statistics, , vol. 38(3), pages 239-266, June.
    • Handle: RePEc:sae:jedbes:v:38:y:2013:i:3:p:239-266
    DOI: 10.3102/1076998612441947
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998612441947
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998612441947?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hong, Guanglei & Raudenbush, Stephen W., 2006. "Evaluating Kindergarten Retention Policy: A Case Study of Causal Inference for Multilevel Observational Data," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 901-910, September.
    2. Rajeev H. Dehejia & Sadek Wahba, 2002. "Propensity Score-Matching Methods For Nonexperimental Causal Studies," The Review of Economics and Statistics, MIT Press, vol. 84(1), pages 151-161, February.
    3. Nadarajah, Saralees & Kotz, Samuel, 2006. "R Programs for Truncated Distributions," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 16(c02).
    4. Evelyn Kitagawa, 1964. "Standardized comparisons in population research," Demography, Springer;Population Association of America (PAA), vol. 1(1), pages 296-315, March.
    5. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    6. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.
    2. Ashley L. Buchanan & Michael G. Hudgens & Stephen R. Cole & Katie R. Mollan & Paul E. Sax & Eric S. Daar & Adaora A. Adimora & Joseph J. Eron & Michael J. Mugavero, 2018. "Generalizing evidence from randomized trials using inverse probability of sampling weights," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(4), pages 1193-1209, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Solomon Asfaw & Silvio Daidone & Benjamin Davis & Josh Dewbre & Alessandro Romeo & Paul Winters & Katia Covarrubias & Habiba Djebbari, 2012. "Analytical Framework for Evaluating the Productive Impact of Cash Transfer Programmes on Household Behaviour – Methodological Guidelines for the From Protection to Production Project," Working Papers 101, International Policy Centre for Inclusive Growth.
    2. Simone Bertoli & Francesca Marchetta, 2014. "Migration, Remittances and Poverty in Ecuador," Journal of Development Studies, Taylor & Francis Journals, vol. 50(8), pages 1067-1089, August.
    3. Patrick Christian Feihle & Jochen Lawrenz, 2017. "The Issuance of German SME Bonds and its Impact on Operating Performance," Schmalenbach Business Review, Springer;Schmalenbach-Gesellschaft, vol. 18(3), pages 227-259, August.
    4. Federico Biagi & Daniele Bondonio & Alberto Martini, 2015. "Counterfactual Impact Evaluation of Enterprise Support Programmes. Evidence from a Decade of Subsidies to Italian Firm," ERSA conference papers ersa15p1619, European Regional Science Association.
    5. Kube, Roland & von Graevenitz, Kathrine & Löschel, Andreas & Massier, Philipp, 2019. "Do voluntary environmental programs reduce emissions? EMAS in the German manufacturing sector," Energy Economics, Elsevier, vol. 84(S1).
    6. K. Poehlmann & R. Helm & O. Mauroner & J. Auburger, 2021. "Corporate spin-offs’ success factors: management lessons from a comparative empirical analysis with research-based spin-offs," Review of Managerial Science, Springer, vol. 15(6), pages 1767-1796, August.
    7. Elaine M. Wolf & Douglas A. Wolf, 2008. "Mixed Results in a Transitional Planning Program for Alternative School Students," Evaluation Review, , vol. 32(2), pages 187-215, April.
    8. Li, Qian & Liu, Shangqun, 2023. "Does alternative data reduce stock price crash risk? Evidence from third-party online sales disclosure in China," International Review of Financial Analysis, Elsevier, vol. 88(C).
    9. Colombo, Massimo G. & D’Adda, Diego & Pirelli, Lorenzo H., 2016. "The participation of new technology-based firms in EU-funded R&D partnerships: The role of venture capital," Research Policy, Elsevier, vol. 45(2), pages 361-375.
    10. Jasjeet Singh Sekhon & Richard D. Grieve, 2012. "A matching method for improving covariate balance in cost‐effectiveness analyses," Health Economics, John Wiley & Sons, Ltd., vol. 21(6), pages 695-714, June.
    11. Alexandru MINEA & Jean-Louis COMBES & Weneyam Hippolyte BALIMA, 2015. "Sovereign Debt Risk in Emerging Countries: Does Inflation Targeting Adoption Make Any Difference?," Working Papers 201504, CERDI.
    12. Shen, Chung-Hua & Wu, Meng-Wen & Chen, Ting-Hsuan & Fang, Hao, 2016. "To engage or not to engage in corporate social responsibility: Empirical evidence from global banking sector," Economic Modelling, Elsevier, vol. 55(C), pages 207-225.
    13. Estifanos, Tafesse Kefyalew & Polyakov, Maksym & Pandit, Ram & Hailu, Atakelty & Burton, Michael, 2020. "The impact of protected areas on the rural households’ incomes in Ethiopia," Land Use Policy, Elsevier, vol. 91(C).
    14. Ragni, Alessandra & Ippolito, Daniel & Masci, Chiara, 2024. "Assessing the impact of hybrid teaching on students’ academic performance via multilevel propensity score-based techniques," Socio-Economic Planning Sciences, Elsevier, vol. 92(C).
    15. Srhoj, Stjepan & Walde, Janette, 2020. "Getting ready for EU Single Market: The effect of export-oriented grant schemes on firm performance," Structural Change and Economic Dynamics, Elsevier, vol. 52(C), pages 279-293.
    16. Burger, Anže & Hogan, Teresa & Kotnik, Patricia & Rao, Sandeep & Sakinç, Mustafa Erdem, 2023. "Does acquisition lead to the growth of high-tech scale-ups? Evidence from Europe," Research in International Business and Finance, Elsevier, vol. 64(C).
    17. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    18. Miguel Minutti‐Meza, 2013. "Does Auditor Industry Specialization Improve Audit Quality?," Journal of Accounting Research, Wiley Blackwell, vol. 51(4), pages 779-817, September.
    19. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    20. Tingting Zhou & Michael R. Elliott & Roderick J. A. Little, 2022. "Addressing Disparities in the Propensity Score Distributions for Treatment Comparisons from Observational Studies," Stats, MDPI, vol. 5(4), pages 1-17, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:38:y:2013:i:3:p:239-266. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.