Dealing with the replication crisis in psychological science: The contribution of Type M and Type S errors

The present work aims to analyze the replicability crisis in psychology with a focus on statistical inference. The main objective is to highlight the risks to beware when performing hypotheses tests in a Frequentist framework. In addition to the classic Type I and Type II errors, two other errors that are not commonly considered are Type M error (magnitude) and Type S error (sign), concerning the size and direction of the effects. The first chapter introduces Null Hypothesis Significance Testing (NHST), the prevalent approach to statistical inference in the social sciences, following a historical perspective and presenting also the approaches of the statisticians Ronald Fisher, Jerzy Neyman and Egon Pearson. The second chapter discusses the replicability crisis in psychology with an analysis of the origins, the factors that contributed to the crisis and solutions proposed for a change of direction. The third chapter analyzes the role of Type M and Type S errors in the replicability crisis. Studies with a high probability of committing these two types of errors could provide estimates of effects that are exaggerated and/or in the wrong direction. Two types of analysis are presented to examine these errors before conducting a study (prospective design analysis) or once the study has already been conducted (retrospective design analysis). The fourth chapter aims to link Type M and Type S errors with the decline effect, which is the observation that the magnitude of effects decreases with repeated replications. Although there may be multiple reasons behind the decline effect, a possible explanation is that the original study overestimated the effect. A case study illustrates how a retrospective design analysis of the original study can provide information on the probability of Type M and Type S errors and give support to the hypothesis of overestimation. The final chapter summarizes the contributions of Type M and Type S errors to the replication crisis and the role of a design analysis in planning studies and analyzing results.