IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2512.06946.html

Testing the Significance of the Difference-in-Differences Coefficient via Doubly Randomised Inference

Author

Listed:
  • Stanis{l}aw Marek Sergiusz Halkiewicz
  • Andrzej Ka{l}u.za

Abstract

This article develops a significance test for the Difference-in-Differences (DiD) estimator based on doubly randomised inference, in which both the treatment and time indicators are permuted to generate an empirical null distribution of the DiD coefficient. Unlike classical $t$-tests or single-margin permutation procedures, the proposed method exploits a substantially enlarged randomization space. We formally characterise this expansion and show that dual randomization increases the number of admissible relabelings by a factor of $\binom{n}{n_T}$, yielding an exponentially richer permutation universe. This combinatorial gain implies a denser and more stable approximation of the null distribution, a result further justified through an information-theoretic (entropy) interpretation. The validity and finite-sample behaviour of the test are examined using multiple empirical datasets commonly analysed in applied economics, including the Indonesian school construction program (INPRES), brand search data, minimum wage reforms, and municipality-level refugee inflows in Greece. Across all settings, doubly randomised inference performs comparably to standard approaches while offering superior small-sample stability and sharper critical regions due to the enlarged permutation space. The proposed procedure therefore provides a robust, nonparametric alternative for assessing the statistical significance of DiD estimates, particularly in designs with limited group sizes or irregular assignment structures.

Suggested Citation

  • Stanis{l}aw Marek Sergiusz Halkiewicz & Andrzej Ka{l}u.za, 2025. "Testing the Significance of the Difference-in-Differences Coefficient via Doubly Randomised Inference," Papers 2512.06946, arXiv.org.
  • Handle: RePEc:arx:papers:2512.06946
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2512.06946
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ivan A. Canay & Joseph P. Romano & Azeem M. Shaikh, 2017. "Randomization Tests Under an Approximate Symmetry Assumption," Econometrica, Econometric Society, vol. 85, pages 1013-1030, May.
    2. Maximilian A M Vermorken, 2011. "GICS or ICB, how different is similar?," Journal of Asset Management, Palgrave Macmillan, vol. 12(1), pages 30-44, April.
    3. Ghadir Asadi, 2020. "Parents' investments in the quality of education: the case of Ghana," Education Economics, Taylor & Francis Journals, vol. 28(6), pages 621-646, November.
    4. Heckman, James & Pinto, Rodrigo & Shaikh, Azeem M., 2024. "Dealing with imperfect randomization: Inference for the highscope perry preschool program," Journal of Econometrics, Elsevier, vol. 243(1).
    5. Guoli Chen & Craig Crossland & Sterling Huang, 2016. "Female board representation and corporate acquisition intensity," Strategic Management Journal, Wiley Blackwell, vol. 37(2), pages 303-313, February.
    6. Dmitry Arkhangelsky & Susan Athey & David A. Hirshberg & Guido W. Imbens & Stefan Wager, 2021. "Synthetic Difference-in-Differences," American Economic Review, American Economic Association, vol. 111(12), pages 4088-4118, December.
    7. Rauf Ahmad & Per Johansson & Mårten Schultzberg, 2024. "Is Fisher inference inferior to Neyman inference for policy analysis?," Statistical Papers, Springer, vol. 65(6), pages 3425-3445, August.
    8. Cattaneo Matias D. & Frandsen Brigham R. & Titiunik Rocío, 2015. "Randomization Inference in the Regression Discontinuity Design: An Application to Party Advantages in the U.S. Senate," Journal of Causal Inference, De Gruyter, vol. 3(1), pages 1-24.
    9. Jingli Ren & Zhibo Cheng & Stefan Siegmund, 2011. "Neutral Operator and Neutral Differential Equation," Abstract and Applied Analysis, John Wiley & Sons, vol. 2011(1).
    10. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, November.
    11. Stefano Bonnini & Getnet Melak Assegie & Kamila Trzcinska, 2024. "Review about the Permutation Approach in Hypothesis Testing," Mathematics, MDPI, vol. 12(17), pages 1-29, August.
    12. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    13. Young, Alwyn, 2019. "Channeling Fisher: randomization tests and the statistical insignificance of seemingly significant experimental results," LSE Research Online Documents on Economics 101401, London School of Economics and Political Science, LSE Library.
    14. Juan M. Villa, 2016. "diff: Simplifying the estimation of difference-in-differences treatment effects," Stata Journal, StataCorp LLC, vol. 16(1), pages 52-71, March.
    15. Anders Fredriksson & Gustavo Magalhães de Oliveira, 2019. "Impact evaluation using Difference-in-Differences," RAUSP Management Journal, Emerald Group Publishing Limited, vol. 54(4), pages 519-532, October.
    16. Esther Duflo, 2001. "Schooling and Labor Market Consequences of School Construction in Indonesia: Evidence from an Unusual Policy Experiment," American Economic Review, American Economic Association, vol. 91(4), pages 795-813, September.
    17. Alwyn Young, 2019. "Channeling Fisher: Randomization Tests and the Statistical Insignificance of Seemingly Significant Experimental Results," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(2), pages 557-598.
    18. MacKinnon, James G. & Webb, Matthew D., 2020. "Randomization inference for difference-in-differences with few treated clusters," Journal of Econometrics, Elsevier, vol. 218(2), pages 435-450.
    19. Jingli Ren & Zhibo Cheng & Stefan Siegmund, 2011. "Neutral Operator and Neutral Differential Equation," Abstract and Applied Analysis, Hindawi, vol. 2011, pages 1-29, August.
    20. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-793, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    2. Bruce E. Hansen, 2025. "Standard Errors for Difference‐in‐Difference Regression," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 40(3), pages 291-309, April.
    3. Dmitry Arkhangelsky & Guido Imbens, 2023. "Causal Models for Longitudinal and Panel Data: A Survey," Papers 2311.15458, arXiv.org, revised Jun 2024.
    4. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    5. Guillaume Basse & Peng Ding & Avi Feller & Panos Toulis, 2024. "Randomization Tests for Peer Effects in Group Formation Experiments," Econometrica, Econometric Society, vol. 92(2), pages 567-590, March.
    6. Luis Alvarez & Bruno Ferman, 2025. "On the relationship between prediction intervals, tests of sharp nulls and inference on realized treatment effects in settings with few treated units," Papers 2506.14998, arXiv.org.
    7. Jeffrey D. Michler & Anna Josephson, 2022. "Recent developments in inference: practicalities for applied economics," Chapters, in: A Modern Guide to Food Economics, chapter 11, pages 235-268, Edward Elgar Publishing.
    8. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    9. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org, revised Feb 2025.
    10. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    11. Luis Alvarez & Bruno Ferman, 2020. "Inference in Difference-in-Differences with Few Treated Units and Spatial Correlation," Papers 2006.16997, arXiv.org, revised Apr 2023.
    12. Johnsen, Åshild A. & Kvaløy, Ola, 2021. "Conspiracy against the public - An experiment on collusion11“People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices.” (Adam," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 94(C).
    13. Kirakozian, Ankinée & Chiappini, Raphaël & Arfaoui, Nabila, 2025. "Nudging employees for greener mobility—A field experiment," Transportation Research Part A: Policy and Practice, Elsevier, vol. 199(C).
    14. David McKenzie, 2025. "Designing and analysing powerful experiments: practical tips for applied researchers," Fiscal Studies, John Wiley & Sons, vol. 46(3), pages 305-322, September.
    15. Rachel Cassidy & Anaya Dam & Wendy Janssens & Umair Kiani & Karlijn Morsink, 2024. "Targeting men, women or both to reduce child marriage," Tinbergen Institute Discussion Papers 22-087/V, Tinbergen Institute, revised 22 Oct 2024.
    16. Jaya Jumrani & J. V. Meenakshi, 2023. "How effective is a fat subsidy? Evidence from edible oil consumption in India," Journal of Agricultural Economics, Wiley Blackwell, vol. 74(2), pages 327-348, June.
    17. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    18. Samuel Nocito & Alessandra Venturini, 2022. "Does Cooperation among Institutions Foster Migrants Inclusion? Evidence from a Case-Study on Financial Literacy in Italy," Working Papers 10/22, Sapienza University of Rome, DISS.
    19. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.
    20. Florian F Gunsilius, 2025. "A primer on optimal transport for causal inference with observational data," Papers 2503.07811, arXiv.org, revised Mar 2025.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2512.06946. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.