IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v89y2024i2d10.1007_s11336-024-09973-6.html
   My bibliography  Save this article

Post-selection Inference in Multiverse Analysis (PIMA): An Inferential Framework Based on the Sign Flipping Score Test

Author

Listed:
  • Paolo Girardi

    (Ca’ Foscari University of Venice)

  • Anna Vesely

    (University of Bologna)

  • Daniël Lakens

    (Eindhoven University of Technology)

  • Gianmarco Altoè

    (University of Padova)

  • Massimiliano Pastore

    (University of Padova)

  • Antonio Calcagnì

    (University of Padova
    GNCS-INdAM RESEARCH GROUP)

  • Livio Finos

    (University of Padova)

Abstract

When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.

Suggested Citation

  • Paolo Girardi & Anna Vesely & Daniël Lakens & Gianmarco Altoè & Massimiliano Pastore & Antonio Calcagnì & Livio Finos, 2024. "Post-selection Inference in Multiverse Analysis (PIMA): An Inferential Framework Based on the Sign Flipping Score Test," Psychometrika, Springer;The Psychometric Society, vol. 89(2), pages 542-568, June.
    • Handle: RePEc:spr:psycho:v:89:y:2024:i:2:d:10.1007_s11336-024-09973-6
    DOI: 10.1007/s11336-024-09973-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-024-09973-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-024-09973-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    2. Flachaire, Emmanuel, 1999. "A better way to bootstrap pairs," Economics Letters, Elsevier, vol. 64(3), pages 257-262, September.
    3. Genovese, Christopher R. & Wasserman, Larry, 2006. "Exceedance Control of the False Discovery Proportion," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1408-1417, December.
    4. Aaditya Ramdas & Rina Foygel Barber & Emmanuel J. Candès & Ryan J. Tibshirani, 2023. "Permutation Tests Using Arbitrary Permutation Distributions," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 85(2), pages 1156-1177, August.
    5. Kerry Dwan & Douglas G Altman & Juan A Arnaiz & Jill Bloom & An-Wen Chan & Eugenia Cronin & Evelyne Decullier & Philippa J Easterbrook & Erik Von Elm & Carrol Gamble & Davina Ghersi & John P A Ioannid, 2008. "Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias," PLOS ONE, Public Library of Science, vol. 3(8), pages 1-31, August.
    6. Jesse Hemerik & Jelle J. Goeman & Livio Finos, 2020. "Robust testing in generalized linear models by sign flipping score contributions," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 82(3), pages 841-864, July.
    7. Colin B. Begg & Jesse A. Berlin, 1988. "Publication Bias: A Problem in Interpreting Medical Data," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 151(3), pages 419-445, May.
    8. Jesse Hemerik & Jelle Goeman, 2018. "Exact testing with random permutations," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 27(4), pages 811-825, December.
    9. Daniele Fanelli, 2012. "Negative results are disappearing from most disciplines and countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 90(3), pages 891-904, March.
    10. Caserotti, Marta & Girardi, Paolo & Rubaltelli, Enrico & Tasso, Alessandra & Lotto, Lorella & Gavaruzzi, Teresa, 2021. "Associations of COVID-19 risk perception with vaccine hesitancy over time for Italian residents," Social Science & Medicine, Elsevier, vol. 272(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. N W Koning & J Hemerik, 2024. "More efficient exact group invariance testing: using a representative subgroup," Biometrika, Biometrika Trust, vol. 111(2), pages 441-458.
    2. Megan L Head & Luke Holman & Rob Lanfear & Andrew T Kahn & Michael D Jennions, 2015. "The Extent and Consequences of P-Hacking in Science," PLOS Biology, Public Library of Science, vol. 13(3), pages 1-15, March.
    3. Robbie C M van Aert & Jelte M Wicherts & Marcel A L M van Assen, 2019. "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-32, April.
    4. David M. Ritzwoller & Joseph P. Romano & Azeem M. Shaikh, 2024. "Randomization Inference: Theory and Applications," Papers 2406.09521, arXiv.org, revised Feb 2025.
    5. Federico Ferraccioli & Laura M. Sangalli & Livio Finos, 2023. "Nonparametric tests for semiparametric regression models," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 32(3), pages 1106-1130, September.
    6. Snyder, Christopher & Zhuo, Ran, 2018. "Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias," MetaArXiv 8vdrh, Center for Open Science.
    7. Brian Fabo & Martina Jancokova & Elisabeth Kempf & Lubos Pastor, 2020. "Fifty Shades of QE: Conflicts of Interest in Economic Research," Working Papers 2020-128, Becker Friedman Institute for Research In Economics.
    8. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    9. Geyer-Klingeberg, Jerome & Hang, Markus & Rathgeber, Andreas, 2020. "Meta-analysis in finance research: Opportunities, challenges, and contemporary applications," International Review of Financial Analysis, Elsevier, vol. 71(C).
    10. Di Pietro Giorgio & European Commission & IZA, 2022. "Studying abroad and earnings: A meta‐analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 36(4), pages 1096-1129, September.
    11. Alex Eble & Peter Boone & Diana Elbourne, 2017. "On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics," The World Bank Economic Review, World Bank, vol. 31(3), pages 687-707.
    12. Chiara Bocci & Annalisa Caloffi & Marco Mariani & Alessandro Sterlacchini, 2023. "Evaluating Public Support to the Investment Activities of Business Firms: A Multilevel Meta-Regression Analysis of Italian Studies," Italian Economic Journal: A Continuation of Rivista Italiana degli Economisti and Giornale degli Economisti, Springer;Società Italiana degli Economisti (Italian Economic Association), vol. 9(1), pages 1-34, March.
    13. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    14. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    15. Michał Krawczyk, 2015. "The Search for Significance: A Few Peculiarities in the Distribution of P Values in Experimental Psychology Literature," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-19, June.
    16. Adam McCloskey & Pascal Michaillat, 2020. "Critical Values Robust to P-hacking," Papers 2005.04141, arXiv.org, revised Dec 2023.
    17. Emilyane de Oliveira Santana Amaral & Sergio Roberto Peres Line, 2021. "Current use of effect size or confidence interval analyses in clinical and biomedical research," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(11), pages 9133-9145, November.
    18. Bruno Ferman & Cristine Pinto & Vitor Possebom, 2020. "Cherry Picking with Synthetic Controls," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 510-532, March.
    19. Stefano DellaVigna & Elizabeth Linos, 2022. "RCTs to Scale: Comprehensive Evidence From Two Nudge Units," Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
    20. Schaub, Jason & Stander, Willem J. & Montgomery, Paul, 2022. "LGBTQ+ Young People’s Health and Well-being Experiences in Out-of-home Social Care: A scoping review," Children and Youth Services Review, Elsevier, vol. 143(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:89:y:2024:i:2:d:10.1007_s11336-024-09973-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.