IDEAS home Printed from https://ideas.repec.org/a/eee/jmvana/v207y2025ics0047259x25000168.html
   My bibliography  Save this article

A review of multivariate permutation tests: Findings and trends

Author

Listed:
  • Arboretti, Rosa
  • Barzizza, Elena
  • Biasetton, Nicoló
  • Disegna, Marta

Abstract

The permutation test is a widely recognized and frequently used nonparametric hypothesis test, notable for its minimal reliance on assumptions compared to parametric tests. It has found applications in many fields, particularly in multivariate analysis. Since its introduction in the 1930s, permutation tests have been extensively examined both theoretically and empirically. This article provides the results of a comprehensive and systematic review of the literature, focusing on different aspects of multivariate permutation tests. Key articles published in international journals from 2010 onwards have been analyzed, classifying them into four main research strands: data, model, test and issues. These strands were further subdivided into more specific categories. The state of the art and significant developments in this field are summarized, followed by a discussion on future research challenges and trends, offering guidance for the design and development on new approaches.

Suggested Citation

  • Arboretti, Rosa & Barzizza, Elena & Biasetton, Nicoló & Disegna, Marta, 2025. "A review of multivariate permutation tests: Findings and trends," Journal of Multivariate Analysis, Elsevier, vol. 207(C).
    • Handle: RePEc:eee:jmvana:v:207:y:2025:i:c:s0047259x25000168
    DOI: 10.1016/j.jmva.2025.105421
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0047259X25000168
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jmva.2025.105421?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Markus Pauly & Edgar Brunner & Frank Konietschke, 2015. "Asymptotic permutation tests in general factorial designs," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 77(2), pages 461-473, March.
    2. William F. Christensen & Brinley N. Zabriskie, 2022. "When Your Permutation Test is Doomed to Fail," The American Statistician, Taylor & Francis Journals, vol. 76(1), pages 53-63, January.
    3. Jin, Ze & Matteson, David S., 2018. "Generalizing distance covariance to measure and test multivariate mutual dependence via complete and incomplete V-statistics," Journal of Multivariate Analysis, Elsevier, vol. 168(C), pages 304-322.
    4. Federico A. Bugni & Joel L. Horowitz, 2021. "Permutation tests for equality of distributions of functional data," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(7), pages 861-877, November.
    5. Hyo-Il Park & Seung-Man Hong, 2010. "A permutation test for multivariate data with grouped components," Journal of Applied Statistics, Taylor & Francis Journals, vol. 37(5), pages 767-778.
    6. Chung, EunYi & Romano, Joseph P., 2016. "Multivariate and multiple permutation tests," Journal of Econometrics, Elsevier, vol. 193(1), pages 76-91.
    7. Ditzhaus, Marc & Smaga, Łukasz, 2022. "Permutation test for the multivariate coefficient of variation in factorial designs," Journal of Multivariate Analysis, Elsevier, vol. 187(C).
    8. Markus Pauly & Maria Umlauft & Ali Ünlü, 2018. "Resampling-Based Inference Methods for Comparing Two Coefficients Alpha," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 203-222, March.
    9. Bonnini, S. & Borghesi, M. & Giacalone, M., 2024. "Semi-parametric approach for modelling overdispersed count data with application to Industry 4.0," Socio-Economic Planning Sciences, Elsevier, vol. 95(C).
    10. Nevalainen, Jaakko & Larocque, Denis & Oja, Hannu & Pörsti, Ilkka, 2010. "Nonparametric Analysis of Clustered Multivariate Data," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 864-872.
    11. Achim Zeileis & Torsten Hothorn, 2013. "A toolbox of permutation tests for structural change," Statistical Papers, Springer, vol. 54(4), pages 931-954, November.
    12. Yinglei Lai & Baolin Wu & Hongyu Zhao, 2011. "A permutation test approach to the choice of size k for the nearest neighbors classifier," Journal of Applied Statistics, Taylor & Francis Journals, vol. 38(10), pages 2289-2302.
    13. Fortunato Pesarin & Luigi Salmaso, 2010. "The permutation testing approach: a review," Statistica, Department of Statistics, University of Bologna, vol. 70(4), pages 481-509.
    14. Arboretti, Rosa & Bonnini, Stefano & Corain, Livio & Salmaso, Luigi, 2014. "A permutation approach for ranking of multivariate populations," Journal of Multivariate Analysis, Elsevier, vol. 132(C), pages 39-57.
    15. Zeng, Chan & Pan, Zhaoxing & MaWhinney, Samantha & Barón, Anna E. & Zerbe, Gary O., 2011. "Permutation and F Distribution of Tests in the Multivariate General Linear Model," The American Statistician, American Statistical Association, vol. 65(1), pages 31-36.
    16. Mirosław Krzyśko & Łukasz Smaga, 2020. "Measuring and Testing Mutual Dependence of Multivariate Functional Data," Statistics in Transition New Series, Polish Statistical Association, vol. 21(3), pages 21-37, September.
    17. Moisés Hernández & Ginés D Guerrero & José M Cecilia & José M García & Alberto Inuggi & Saad Jbabdi & Timothy E J Behrens & Stamatios N Sotiropoulos, 2013. "Accelerating Fibre Orientation Estimation from Diffusion Weighted Magnetic Resonance Imaging Using GPUs," PLOS ONE, Public Library of Science, vol. 8(4), pages 1-13, April.
    18. Marc Ditzhaus & Roland Fried & Markus Pauly, 2021. "QANOVA: quantile-based permutation methods for general factorial designs," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 30(4), pages 960-979, December.
    19. Dylan M Nielson & Per B Sederberg, 2017. "MELD: Mixed effects for large datasets," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-21, August.
    20. Chung, EunYi & Olivares, Mauricio, 2021. "Permutation test for heterogeneous treatment effects with a nuisance parameter," Journal of Econometrics, Elsevier, vol. 225(2), pages 148-174.
    21. Friedrich, Sarah & Brunner, Edgar & Pauly, Markus, 2017. "Permuting longitudinal data in spite of the dependencies," Journal of Multivariate Analysis, Elsevier, vol. 153(C), pages 255-265.
    22. Lovato, Ilenia & Pini, Alessia & Stamm, Aymeric & Vantini, Simone, 2020. "Model-free two-sample test for network-valued data," Computational Statistics & Data Analysis, Elsevier, vol. 144(C).
    23. Fortunato Pesarin & Luigi Salmaso, 2010. "Finite-sample consistency of combination-based permutation tests with application to repeated measures designs," Journal of Nonparametric Statistics, Taylor & Francis Journals, vol. 22(5), pages 669-684.
    24. Ma, Zichen & Hanson, Timothy E., 2020. "Bayesian nonparametric test for independence between random vectors," Computational Statistics & Data Analysis, Elsevier, vol. 149(C).
    25. Luigi Salmaso, 2015. "Combination-Based Permutation Tests: Equipower Property and Power Behavior in Presence of Correlation," Communications in Statistics - Theory and Methods, Taylor & Francis Journals, vol. 44(24), pages 5225-5239, December.
    26. Rosa Arboretti & Riccardo Ceccato & Livio Corain & Fabrizio Ronchi & Luigi Salmaso, 2018. "Multivariate small sample tests for two-way designs with applications to industrial statistics," Statistical Papers, Springer, vol. 59(4), pages 1483-1503, December.
    27. Rosa Arboretti & Fortunato Pesarin & Luigi Salmaso, 2021. "A unified approach to permutation testing for equivalence," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 30(3), pages 1033-1052, September.
    28. Seongoh Park & Johan Lim & Xinlei Wang & Sanghan Lee, 2019. "Permutation based testing on covariance separability," Computational Statistics, Springer, vol. 34(2), pages 865-883, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ditzhaus, Marc & Smaga, Łukasz, 2022. "Permutation test for the multivariate coefficient of variation in factorial designs," Journal of Multivariate Analysis, Elsevier, vol. 187(C).
    2. Stefano Bonnini & Getnet Melak Assegie & Kamila Trzcinska, 2024. "Review about the Permutation Approach in Hypothesis Testing," Mathematics, MDPI, vol. 12(17), pages 1-29, August.
    3. Dennis Dobler & Markus Pauly, 2018. "Bootstrap- and permutation-based inference for the Mann–Whitney effect for right-censored and tied data," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 27(3), pages 639-658, September.
    4. Sattler, Paavo & Zimmermann, Georg, 2024. "Choice of the hypothesis matrix for using the Wald-type-statistic," Statistics & Probability Letters, Elsevier, vol. 208(C).
    5. Friedrich, Sarah & Pauly, Markus, 2018. "MATS: Inference for potentially singular and heteroscedastic MANOVA," Journal of Multivariate Analysis, Elsevier, vol. 165(C), pages 166-179.
    6. Friedrich, Sarah & Brunner, Edgar & Pauly, Markus, 2017. "Permuting longitudinal data in spite of the dependencies," Journal of Multivariate Analysis, Elsevier, vol. 153(C), pages 255-265.
    7. Young, Alwyn, 2024. "Asymptotically robust permutation-based randomization confidence intervals for parametric OLS regression," LSE Research Online Documents on Economics 120933, London School of Economics and Political Science, LSE Library.
    8. Federico Bugni & Jackson Bunting & Muyang Ren, 2024. "Marginal homogeneity tests with panel data," Papers 2408.15862, arXiv.org.
    9. Jesse Hemerik & Jelle J. Goeman & Livio Finos, 2020. "Robust testing in generalized linear models by sign flipping score contributions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 841-864, July.
    10. Baumeister, Marléne & Ditzhaus, Marc & Pauly, Markus, 2024. "Quantile-based MANOVA: A new tool for inferring multivariate data in factorial designs," Journal of Multivariate Analysis, Elsevier, vol. 199(C).
    11. EunYi Chung & Mauricio Olivares, 2025. "Quantile‐Based Test for Heterogeneous Treatment Effects," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 40(1), pages 3-17, January.
    12. Shreyashi Basak & Markus Pauly & Somesh Kumar, 2024. "Adaptive tests for ANOVA in Fisher–von Mises–Langevin populations under heteroscedasticity," Computational Statistics, Springer, vol. 39(2), pages 433-459, April.
    13. Young, Alwyn, 2024. "Asymptotically robust permutation-based randomization confidence intervals for parametric OLS regression," European Economic Review, Elsevier, vol. 163(C).
    14. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org, revised Feb 2025.
    15. Konietschke, Frank & Bathke, Arne C. & Harrar, Solomon W. & Pauly, Markus, 2015. "Parametric and nonparametric bootstrap methods for general MANOVA," Journal of Multivariate Analysis, Elsevier, vol. 140(C), pages 291-301.
    16. Jung, Sungkyu & Sen, Arusharka & Marron, J.S., 2012. "Boundary behavior in High Dimension, Low Sample Size asymptotics of PCA," Journal of Multivariate Analysis, Elsevier, vol. 109(C), pages 190-203.
    17. Roman Tikhonov & Aleksey Masyutin & Vadim Anpilogov, 2021. "The Relationship Between the Financial Performance of Banks and the Quality of Credit Scoring Models," Russian Journal of Money and Finance, Bank of Russia, vol. 80(2), pages 76-95, June.
    18. Federico Ferraccioli & Laura M. Sangalli & Livio Finos, 2023. "Nonparametric tests for semiparametric regression models," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 32(3), pages 1106-1130, September.
    19. Dickhaus, Thorsten & Sirotko-Sibirskaya, Natalia, 2019. "Simultaneous statistical inference in dynamic factor models: Chi-square approximation and model-based bootstrap," Computational Statistics & Data Analysis, Elsevier, vol. 129(C), pages 30-46.
    20. Julius Owusu, 2023. "Randomization Inference of Heterogeneous Treatment Effects under Network Interference," Papers 2308.00202, arXiv.org, revised Jun 2025.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jmvana:v:207:y:2025:i:c:s0047259x25000168. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/622892/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.