Analyzing nested experimental designs—A user-friendly resampling method to determine experimental significance

While hierarchical experimental designs are near-ubiquitous in neuroscience and biomedical research, researchers often do not take the structure of their datasets into account while performing statistical hypothesis tests. Resampling-based methods are a flexible strategy for performing these analyses but are difficult due to the lack of open-source software to automate test construction and execution. To address this, we present Hierarch, a Python package to perform hypothesis tests and compute confidence intervals on hierarchical experimental designs. Using a combination of permutation resampling and bootstrap aggregation, Hierarch can be used to perform hypothesis tests that maintain nominal Type I error rates and generate confidence intervals that maintain the nominal coverage probability without making distributional assumptions about the dataset of interest. Hierarch makes use of the Numba JIT compiler to reduce p-value computation times to under one second for typical datasets in biomedical research. Hierarch also enables researchers to construct user-defined resampling plans that take advantage of Hierarch’s Numba-accelerated functions.Author summary: An important step in analyzing experimental data is quantifying uncertainty in the experimenter’s conclusions. One mechanism for doing so is by using a statistical hypothesis test, which allows the experimenter to control what percentage of the time they make erroneous conclusions over the course of their career. Biological experimental designs often have hierarchical data-gathering schemes that traditional hypothesis tests are not well-suited for (for example, an experimenter may make measurements of several tissue samples that were collected from subjects who were given a treatment). While traditional tests can be adapted to hierarchical experimental designs, we propose a simple resampling-based hypothesis test that applies to a variety of experimental designs while maintaining control over error rate. In this manuscript, we describe Hierarch, the Python package that enables users to carry out this test and validate it under several conditions.