IDEAS home Printed from https://ideas.repec.org/p/zbw/i4rdps/18.html
   My bibliography  Save this paper

The Necessity of Construct and External Validity for Generalized Causal Claims

Author

Listed:
  • Esterling, Kevin M.
  • Brady, David
  • Schwitzgebel, Eric

Abstract

The Credibility Revolution advances quantitative research designs intended to identify causal effects from observed data. The ensuing emphasis on internal validity however has enabled the neglect of construct and external validity. This article develops a framework we call causal specification. The framework formally demonstrates the joint necessity of assumptions regarding internal, construct and external validity for causal generalization. Indeed, the lack of any of the three types of validity undermines the Credibility Revolution's own goal to understand causality deductively. Without assumptions regarding construct validity, one cannot accurately label the cause or outcome. Without assumptions regarding external validity, one cannot label the conditions enabling the cause to have an effect. These assumptions ultimately are founded on qualitative and theoretical understandings of a causal process. As a result, causal specification clarifies the central role of qualitative research in underwriting deductive understandings of causality in quantitative research.

Suggested Citation

  • Esterling, Kevin M. & Brady, David & Schwitzgebel, Eric, 2023. "The Necessity of Construct and External Validity for Generalized Causal Claims," I4R Discussion Paper Series 18, The Institute for Replication (I4R).
    • Handle: RePEc:zbw:i4rdps:18
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/268605/1/I4R-DP018.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Keele, Luke & Minozzi, William, 2013. "How Much Is Minnesota Like Wisconsin? Assumptions and Counterfactuals in Causal Inference with Observational Data," Political Analysis, Cambridge University Press, vol. 21(2), pages 193-216, April.
    2. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    3. Keele, Luke, 2015. "The Statistics of Causal Inference: A View from Political Methodology," Political Analysis, Cambridge University Press, vol. 23(3), pages 313-335, July.
    4. Adcock, Robert & Collier, David, 2001. "Measurement Validity: A Shared Standard for Qualitative and Quantitative Research," American Political Science Review, Cambridge University Press, vol. 95(3), pages 529-546, September.
    5. Gerber, Alan S. & Green, Donald P., 2000. "The Effects of Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment," American Political Science Review, Cambridge University Press, vol. 94(3), pages 653-663, September.
    6. David Card, 2022. "Design-Based Research in Empirical Microeconomics," American Economic Review, American Economic Association, vol. 112(6), pages 1773-1781, June.
    7. Sean Gailmard, 2021. "Theory, History, and Political Economy," Journal of Historical Political Economy, now publishers, vol. 1(1), pages 69-104, June.
    8. Joseph Henrich & Steve J. Heine & Ara Norenzayan, 2010. "The Weirdest People in the World?," RatSWD Working Papers 139, German Data Forum (RatSWD).
    9. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    10. Abhijit V. Banerjee & Esther Duflo, 2009. "The Experimental Approach to Development Economics," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 151-178, May.
    11. Deaton, Angus & Cartwright, Nancy, 2018. "Understanding and misunderstanding randomized controlled trials," Social Science & Medicine, Elsevier, vol. 210(C), pages 2-21.
    12. Michael J. Weiss & Howard S. Bloom & Thomas Brock, 2014. "A Conceptual Framework For Studying The Sources Of Variation In Program Effects," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(3), pages 778-808, June.
    13. Pearl Judea, 2019. "Sufficient Causes: On Oxygen, Matches, and Fires," Journal of Causal Inference, De Gruyter, vol. 7(2), pages 1-8, September.
    14. Pearl Judea, 2019. "Sufficient Causes: On Oxygen, Matches, and Fires," Journal of Causal Inference, De Gruyter, vol. 7(2), pages 1-8, September.
    15. Andrew Gelman & Guido Imbens, 2013. "Why ask Why? Forward Causal Inference and Reverse Causal Questions," NBER Working Papers 19614, National Bureau of Economic Research, Inc.
    16. Thomas D. Cook, 2014. "Generalizing Causal Knowledge In The Policy Sciences: External Validity As A Task Of Both Multiattribute Representation And Multiattribute Extrapolation," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(2), pages 527-536, March.
    17. Elizabeth A. Stuart & Stephen R. Cole & Catherine P. Bradshaw & Philip J. Leaf, 2011. "The use of propensity scores to assess the generalizability of results from randomized trials," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(2), pages 369-386, April.
    18. Eva Vivalt, 2020. "How Much Can We Generalize From Impact Evaluations?," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 3045-3089.
    19. Dafoe, Allan & Zhang, Baobao & Caughey, Devin, 2018. "Information Equivalence in Survey Experiments," Political Analysis, Cambridge University Press, vol. 26(4), pages 399-416, October.
    20. Gerber, Alan S. & Green, Donald P. & Larimer, Christopher W., 2008. "Social Pressure and Voter Turnout: Evidence from a Large-Scale Field Experiment," American Political Science Review, Cambridge University Press, vol. 102(1), pages 33-48, February.
    21. Guala,Francesco, 2005. "The Methodology of Experimental Economics," Cambridge Books, Cambridge University Press, number 9780521618618.
    22. Lant Pritchett, Justin Sandefur, 2013. "Context Matters for Size: Why External Validity Claims and Development Practice Don't Mix-Working Paper 336," Working Papers 336, Center for Global Development.
    23. Kosuke Imai & Gary King & Elizabeth A. Stuart, 2008. "Misunderstandings between experimentalists and observationalists about causal inference," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 171(2), pages 481-502, April.
    24. Alan Gerber & Donald Green, 2000. "The effects of canvassing, direct mail, and telephone contact on voter turnout: A field experiment," Natural Field Experiments 00248, The Field Experiments Website.
    25. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ankel-Peters, Jörg & Schmidt, Christoph M., 2023. "Rural electrification, the credibility revolution, and the limits of evidence-based policy," Ruhr Economic Papers 1051, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Esterling, Kevin & Brady, David & Schwitzgebel, Eric, 2021. "The Necessity of Construct and External Validity for Generalized Causal Claims," OSF Preprints 2s8w5, Center for Open Science.
    2. Denis Fougère & Nicolas Jacquemet, 2020. "Policy Evaluation Using Causal Inference Methods," SciencePo Working papers Main hal-03455978, HAL.
    3. Naoki Egami & Erin Hartman, 2021. "Covariate selection for generalizing experimental results: Application to a large‐scale development program in Uganda," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(4), pages 1524-1548, October.
    4. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    5. Ashis Das & Jed Friedman & Eeshani Kandpal, 2018. "Does involvement of local NGOs enhance public service delivery? Cautionary evidence from a malaria‐prevention program in India," Health Economics, John Wiley & Sons, Ltd., vol. 27(1), pages 172-188, January.
    6. León, Gianmarco, 2017. "Turnout, political preferences and information: Experimental evidence from Peru," Journal of Development Economics, Elsevier, vol. 127(C), pages 56-71.
    7. Alan Gerber & Mitchell Hoffman & John Morgan & Collin Raymond, 2020. "One in a Million: Field Experiments on Perceived Closeness of the Election and Voter Turnout," American Economic Journal: Applied Economics, American Economic Association, vol. 12(3), pages 287-325, July.
    8. Alberto Chong & Gianmarco León‐Ciliotta & Vivian Roza & Martín Valdivia & Gabriela Vega, 2019. "Urbanization Patterns, Information Diffusion, and Female Voting in Rural Paraguay," American Journal of Political Science, John Wiley & Sons, vol. 63(2), pages 323-341, April.
    9. Valentina A. Bali & Lindon J. Robison & Richard Winder, 2020. "What Motivates People to Vote? The Role of Selfishness, Duty, and Social Motives When Voting," SAGE Open, , vol. 10(4), pages 21582440209, October.
    10. Bogliacino, Francesco & Codagnone, Cristiano, 2021. "Microfoundations, behaviour, and evolution: Evidence from experiments," Structural Change and Economic Dynamics, Elsevier, vol. 56(C), pages 372-385.
    11. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    12. Elizabeth A. Stuart & Anna Rhodes, 2017. "Generalizing Treatment Effect Estimates From Sample to Population: A Case Study in the Difficulties of Finding Sufficient Data," Evaluation Review, , vol. 41(4), pages 357-388, August.
    13. Judith Favereau & Nicolas Brisset, 2016. "Randomization of What? Moving from Libertarian to "Democratic Paternalism"," GREDEG Working Papers 2016-34, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    14. Alberto Chong & Gianmarco León & Vivian Roza & Martin Valdivia & Gabriela Vega, 2017. "Urbanization patterns, social interactions and female voting in rural Paraguay," Economics Working Papers 1589, Department of Economics and Business, Universitat Pompeu Fabra.
    15. Bratton, Michael, 2013. "Measuring Government Performance in Public Opinion Surveys in Africa: Towards Experiments?," WIDER Working Paper Series 023, World Institute for Development Economic Research (UNU-WIDER).
    16. Davide Cantoni & David Y Yang & Noam Yuchtman & Y Jane Zhang, 2019. "Protests as Strategic Games: Experimental Evidence from Hong Kong's Antiauthoritarian Movement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(2), pages 1021-1077.
    17. Felix Koelle & Tom Lane & Daniele Nosenzo & Chris Starmer, 2017. "Nudging the electorate: what works and why?," Discussion Papers 2017-16, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    18. Jordan Adamson & Lucas Rentschler, 2023. "Criminal justice from a public choice perspective: an introduction to the special issue," Public Choice, Springer, vol. 196(3), pages 223-227, September.
    19. Jens Ludwig & Jeffrey R. Kling & Sendhil Mullainathan, 2011. "Mechanism Experiments and Policy Evaluations," Journal of Economic Perspectives, American Economic Association, vol. 25(3), pages 17-38, Summer.
    20. Grácio, Matilde & Vicente, Pedro C., 2021. "Information, get-out-the-vote messages, and peer influence: Causal effects on political behavior in Mozambique," Journal of Development Economics, Elsevier, vol. 151(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:i4rdps:18. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://www.i4replication.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.