IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v51y2022i4p1947-1968.html
   My bibliography  Save this article

The probability of a robust inference for internal validity

Author

Listed:
  • Tenglong Li
  • Ken Frank

Abstract

The internal validity of observational study is often subject to debate. In this study, we define the counterfactuals as the unobserved sample and intend to quantify its relationship with the null hypothesis statistical testing (NHST). We propose the probability of a robust inference for internal validity, that is, the PIV, as a robustness index of causal inference. Formally, the PIV is the probability of rejecting the null hypothesis again based on both the observed sample and the counterfactuals, provided the same null hypothesis has already been rejected based on the observed sample. Under either frequentist or Bayesian framework, one can bound the PIV of an inference based on his bounded belief about the counterfactuals, which is often needed when the unconfoundedness assumption is dubious. The PIV is equivalent to statistical power when the NHST is thought to be based on both the observed sample and the counterfactuals. We summarize the process of evaluating internal validity with the PIV into a six-step procedure and illustrate it with an empirical example.

Suggested Citation

  • Tenglong Li & Ken Frank, 2022. "The probability of a robust inference for internal validity," Sociological Methods & Research, , vol. 51(4), pages 1947-1968, November.
    • Handle: RePEc:sae:somere:v:51:y:2022:i:4:p:1947-1968
    DOI: 10.1177/0049124120914922
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124120914922
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124120914922?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Tyler J. VanderWeele, 2008. "Sensitivity Analysis: Distributional Assumptions and Confounding Assumptions," Biometrics, The International Biometric Society, vol. 64(2), pages 645-649, June.
    2. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    3. Donald B. Rubin, 2005. "Causal Inference Using Potential Outcomes: Design, Modeling, Decisions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 322-331, March.
    4. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    5. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tenglong Li & Kenneth A. Frank & Mingming Chen, 2024. "A Conceptual Framework for Quantifying the Robustness of a Regression-Based Causal Inference in Observational Study," Mathematics, MDPI, vol. 12(3), pages 1-14, January.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Tenglong Li & Kenneth A. Frank & Mingming Chen, 2024. "A Conceptual Framework for Quantifying the Robustness of a Regression-Based Causal Inference in Observational Study," Mathematics, MDPI, vol. 12(3), pages 1-14, January.
    2. Tenglong Li & Kenneth A. Frank, 2020. "The probability of a robust inference for internal validity and its applications in regression models," Papers 2005.12784, arXiv.org.
    3. Tenglong Li & Kenneth A. Frank, 2019. "On the probability of a causal inference is robust for internal validity," Papers 1906.08726, arXiv.org.
    4. Guido W. Imbens, 2020. "Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics," Journal of Economic Literature, American Economic Association, vol. 58(4), pages 1129-1179, December.
    5. Zhexiao Lin & Peng Ding & Fang Han, 2023. "Estimation Based on Nearest Neighbor Matching: From Density Ratio to Average Treatment Effect," Econometrica, Econometric Society, vol. 91(6), pages 2187-2217, November.
    6. Tenglong Li & Jordan Lawson, 2021. "A generalized bootstrap procedure of the standard error and confidence interval estimation for inverse probability of treatment weighting," Papers 2109.00171, arXiv.org.
    7. Luke Keele & Stephen O’Neill & Richard Grieve, 2021. "Comparing the Performance of Statistical Adjustment Methods by Recovering the Experimental Benchmark from the REFLUX Trial," Medical Decision Making, , vol. 41(3), pages 340-353, April.
    8. Tang, Chuan & Heintzelman, Martin D. & Holsen, Thomas M., 2018. "Mercury pollution, information, and property values," Journal of Environmental Economics and Management, Elsevier, vol. 92(C), pages 418-432.
    9. Brian G. Vegetabile & Daniel L. Gillen & Hal S. Stern, 2020. "Optimally balanced Gaussian process propensity scores for estimating treatment effects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(1), pages 355-377, January.
    10. Matt Taddy & Matt Gardner & Liyun Chen & David Draper, 2016. "A Nonparametric Bayesian Analysis of Heterogenous Treatment Effects in Digital Experimentation," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 34(4), pages 661-672, October.
    11. Franz R. Hahn & Werner Hölzl & Claudia Kwapil, 2016. "The Credit Channel and the Role of Monetary Policy Before, During and After the Global Financial Crisis. A Micro Data Approach to the Analysis of Bank-firm Relationships," WIFO Studies, WIFO, number 59233, Juni.
    12. Hang Miao & Kui Zhao & Zhun Wang & Linbo Jiang & Quanhui Jia & Yanming Fang & Quan Yu, 2020. "Intelligent Credit Limit Management in Consumer Loans Based on Causal Inference," Papers 2007.05188, arXiv.org.
    13. Peter Z. Schochet, "undated". "Statistical Theory for the RCT-YES Software: Design-Based Causal Inference for RCTs," Mathematica Policy Research Reports a0c005c003c242308a92c02dc, Mathematica Policy Research.
    14. Pablo Geraldo Bast'ias, 2024. "Credible causal inference beyond toy models," Papers 2402.11659, arXiv.org.
    15. Zhexiao Lin & Peng Ding & Fang Han, 2021. "Estimation based on nearest neighbor matching: from density ratio to average treatment effect," Papers 2112.13506, arXiv.org.
    16. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-dimensional econometrics and regularized GMM," CeMMAP working papers CWP35/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    17. Caloffi, Annalisa & Freo, Marzia & Ghinoi, Stefano & Mariani, Marco & Rossi, Federica, 2022. "Assessing the effects of a deliberate policy mix: The case of technology and innovation advisory services and innovation vouchers," Research Policy, Elsevier, vol. 51(6).
    18. Shanike J. Smart & Solomon W. Polachek, 2024. "COVID-19 vaccine and risk-taking," Journal of Risk and Uncertainty, Springer, vol. 68(1), pages 25-49, February.
    19. Florian Gunsilius & Yuliang Xu, 2021. "Matching for causal effects via multimarginal unbalanced optimal transport," Papers 2112.04398, arXiv.org, revised Jul 2022.
    20. Zhengyuan Zhou & Susan Athey & Stefan Wager, 2023. "Offline Multi-Action Policy Learning: Generalization and Optimization," Operations Research, INFORMS, vol. 71(1), pages 148-183, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:51:y:2022:i:4:p:1947-1968. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.