IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/2dgrp_v1.html

Manipulation Tests in Regression Discontinuity Design: The Need for Equivalence Testing

Author

Listed:
  • Fitzgerald, Jack

    (Vrije Universiteit Amsterdam)

Abstract

Researchers applying regression discontinuity design (RDD) often test for endogenous running variable (RV) manipulation around treatment cutoffs, but misinterpret statistically insignificant RV manipulation as evidence of negligible RV manipulation. I introduce novel procedures that can provide statistically significant evidence that RV manipulation around a cutoff is bounded beneath practically negligible levels. The procedures augment classic RV density tests with an equivalence testing framework, along with bootstrap methods for (cluster-)robust inference. I apply these procedures to replication data from 36 RDD publications, conducting 45 equivalence-based RV manipulation tests. Over 44% of RV density discontinuities at the cutoff cannot be significantly bounded beneath a 50% upward jump. Obtaining equivalence testing failure rates beneath 5% requires arguing that a 350% upward RV density jump at the cutoff is practically equal to zero. My results imply that meaningful RV manipulation around treatment cutoffs cannot be ruled out in many published RDD papers, and that standard tests frequently misclassify the practical significance of RV manipulation. I provide research guidelines and help researchers conduct more credible equivalence-based manipulation testing in future RDD research. The lddtest estimation routine is available in R, Stata, and Python.

Suggested Citation

  • Fitzgerald, Jack, 2025. "Manipulation Tests in Regression Discontinuity Design: The Need for Equivalence Testing," MetaArXiv 2dgrp_v1, Center for Open Science.
    • Handle: RePEc:osf:metaar:2dgrp_v1
    DOI: 10.31219/osf.io/2dgrp_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/67a368d5b0f5b6e473af4482/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/2dgrp_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Andrew C. Eggers & Anthony Fowler & Jens Hainmueller & Andrew B. Hall & James M. Snyder, 2015. "On the Validity of the Regression Discontinuity Design for Estimating Electoral Effects: New Evidence from Over 40,000 Close Races," American Journal of Political Science, John Wiley & Sons, vol. 59(1), pages 259-274, January.
    3. Stommes, Drew & Aronow, P. M. & Sävje, Fredrik, 2023. "On the Reliability of Published Findings Using the Regression Discontinuity Design in Political Science," I4R Discussion Paper Series 22, The Institute for Replication (I4R).
    4. Caughey, Devin & Sekhon, Jasjeet S., 2011. "Elections and the Regression Discontinuity Design: Lessons from Close U.S. House Races, 1942–2008," Political Analysis, Cambridge University Press, vol. 19(4), pages 385-408.
    5. Bugni, Federico A. & Canay, Ivan A., 2021. "Testing continuity of a density via g-order statistics in the regression discontinuity design," Journal of Econometrics, Elsevier, vol. 221(1), pages 138-159.
    6. Erin Hartman & F. Daniel Hidalgo, 2018. "An Equivalence Approach to Balance and Placebo Tests," American Journal of Political Science, John Wiley & Sons, vol. 62(4), pages 1000-1013, October.
    7. Gaku Igarashi, 2023. "A nonparametric discontinuity test of density using a beta kernel," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 35(2), pages 323-354, April.
    8. Jun Ma & Hugo Jales & Zhengfei Yu, 2020. "Minimum Contrast Empirical Likelihood Inference of Discontinuity in Density," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(4), pages 934-950, October.
    9. François Gerard & Miikka Rokkanen & Christoph Rothe, 2020. "Bounds on treatment effects in regression discontinuity designs with a manipulated running variable," Quantitative Economics, Econometric Society, vol. 11(3), pages 839-870, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jin‐young Choi & Myoung‐jae Lee, 2023. "Score manipulation, density continuity and intent‐to‐treat effect for regression discontinuity," Pacific Economic Review, Wiley Blackwell, vol. 28(4), pages 552-569, October.
    2. Matias D. Cattaneo & Rocío Titiunik, 2022. "Regression Discontinuity Designs," Annual Review of Economics, Annual Reviews, vol. 14(1), pages 821-851, August.
    3. Bartnicki, Sławomir & Alimowski, Maciej & Górecki, Maciej A., 2022. "The anomalous electoral advantage: Evidence from over 17,000 mayoral candidacies in Poland," European Journal of Political Economy, Elsevier, vol. 72(C).
    4. Francisco Pino, 2014. "Is There Gender Bias Among Voters ?Evidence from the Chilean Congressional Elections," Working Papers ECARES ECARES 2014-53, ULB -- Universite Libre de Bruxelles.
    5. Gouvêa, Raphael & Girardi, Daniele, 2021. "Partisanship and local fiscal policy: Evidence from Brazilian cities," Journal of Development Economics, Elsevier, vol. 150(C).
    6. Brown, Craig O., 2020. "Economic leadership and growth," Journal of Monetary Economics, Elsevier, vol. 116(C), pages 298-333.
    7. Ari Hyytinen & Jaakko Meriläinen & Tuukka Saarimaa & Otto Toivanen & Janne Tukiainen, 2018. "When does regression discontinuity design work? Evidence from random election outcomes," Quantitative Economics, Econometric Society, vol. 9(2), pages 1019-1051, July.
    8. Gregory J. Wawro & Ira Katznelson, 2020. "American political development and new challenges of causal inference," Public Choice, Springer, vol. 185(3), pages 299-314, December.
    9. Yuta Okamoto & Yuuki Ozaki, 2024. "On Extrapolation of Treatment Effects in Multiple-Cutoff Regression Discontinuity Designs," Papers 2412.04265, arXiv.org, revised Sep 2025.
    10. Takuya Ishihara & Masayuki Sawada, 2020. "Manipulation-Robust Regression Discontinuity Designs," Papers 2009.07551, arXiv.org, revised Sep 2024.
    11. Lahoti, Rahul & Sahoo, Soham, 2020. "Are educated leaders good for education? Evidence from India," Journal of Economic Behavior & Organization, Elsevier, vol. 176(C), pages 42-62.
    12. Thushyanthan Baskaran & Sonia Bhalotra & Brian Min & Yogesh Uppal, 2024. "Women legislators and economic performance," Journal of Economic Growth, Springer, vol. 29(2), pages 151-214, June.
    13. Leandro De Magalhaes & Isabel Spirgel-Sinclair, 2021. "Could Regression Discontinuity estimates of incumbency e ects help monitor parliamentary elections? Evidence from Malawi," Bristol Economics Discussion Papers 21/741, School of Economics, University of Bristol, UK.
    14. Quoc-Anh Do & Yen-Teik Lee & Bang Dang Nguyen, 2016. "Directors as Connectors: The Impact of the External Networks of Directors on Firms," Working Papers hal-03393196, HAL.
    15. Hsu, Yu-Chin & Shiu, Ji-Liang & Wan, Yuanyuan, 2024. "Testing identification conditions of LATE in fuzzy regression discontinuity designs," Journal of Econometrics, Elsevier, vol. 241(1).
    16. Brian Beach & Daniel B. Jones, 2017. "Gridlock: Ethnic Diversity in Government and the Provision of Public Goods," American Economic Journal: Economic Policy, American Economic Association, vol. 9(1), pages 112-136, February.
    17. Magdalena Delaporte & Francisco Pino, 2022. "Female Political Representation and Violence Against Women: Evidence from Brazil," Working Papers wp534, University of Chile, Department of Economics.
    18. Yoichi Arai & Yu‐Chin Hsu & Toru Kitagawa & Ismael Mourifié & Yuanyuan Wan, 2022. "Testing identifying assumptions in fuzzy regression discontinuity designs," Quantitative Economics, Econometric Society, vol. 13(1), pages 1-28, January.
    19. Andrew Bertoli & Allan Dafoe & Robert F. Trager, 2019. "Is There a War Party? Party Change, the Left–Right Divide, and International Conflict," Journal of Conflict Resolution, Peace Science Society (International), vol. 63(4), pages 950-975, April.
    20. Quoc-Anh Do & Yen-Teik Lee & Bang Dang Nguyen, 2013. "Political Connections and Firm Value: Evidence from the Regression Discontinuity Design of Close Gubernatorial Elections," Sciences Po Economics Publications (main) hal-03460972, HAL.

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C87 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Econometric Software
    • P00 - Political Economy and Comparative Economic Systems - - General - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:2dgrp_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.