Multi-Studies: A Novel Approach to Addressing Irreplicability in RCTs

This chapter proposes a novel approach to addressing irreplicability encountered in randomised controlled trials (RCTs). Irreplicability affects, to different degrees, all fields of science. Some of the most common explanations for non-replicability are low sample size, low statistical power, p-hacking, publication bias and HARKing. Such issues face some studies but not others. Focusing on RCTs, this chapter highlights that, while it is well-recognised that many types of bias and other types of constraint generally face studies across the biomedical, behavioural and social sciences, they are not commonly discussed collectively. In response to this gap in the literature on RCTs, we shall consider a novel approach to addressing non-replicability. The chapter helps explain how a degree of inevitable variation between the outcomes of different studies is driven by a combined set of biases and constraints underlying the scientific process that relate to methods, people, contexts, and time points at which data are collected in studies. It outlines how causal evidence in such studies is inferred from estimated statistical results that are the outcome of multiple complex processes: sampling, randomising, blinding, controlling, delivering treatments, and so on. These processes involve many actors making many decisions at different steps when designing, implementing and analysing studies within a particular context and time period. Correspondingly, the chapter considers the influence of the combined set of constraints on the quality and replicability of RCTs, and how to better tackle this issue. This entails integrating comparability of groups, treatments, contexts, etc. within a single study—called here a multi-study. Implications are then drawn for practitioners.