Despite the importance of excess mortality estimates for assessing the scale of a pandemic and informing policy decisions, little attention has been paid to the statistical properties of the methods used to generate them. In this paper, we show that the empirical coverage of prediction intervals for 24-month cumulative excess mortality produced by two of the most influential approaches falls well below the nominal 95% level. The World Health Organization (WHO) methodology, for example, exhibits coverage as low as 30%. We propose two models for estimating excess mortality—one frequentist and one Bayesian—both of which are variants of the WHO model and produce prediction intervals with empirical coverage close to the nominal levels. Moreover, we introduce a simple correction method that can be applied to any model for which pre-pandemic death-count time series are observed. The correction calibrates prediction intervals using the historical validation environment, allowing researchers to address undercoverage on average across units and periods. We provide two concrete examples illustrating the practical importance of working with prediction intervals with correct coverage. First, we focus on the underreporting gap, defined as excess deaths minus reported COVID-19 deaths. Using the WHO method, uncorrected estimates suggest that 41 of 51 U.S. jurisdictions and 13 of 30 European countries experienced a significantly positive underreporting gap. In contrast, applying our correction reduces these numbers to 22 and 6, respectively. These findings indicate that the statistical evidence for a positive gap between excess deaths and reported COVID-19 deaths is weaker and far less widespread than suggested by methods with overly narrow prediction intervals. Second, we show that rankings of groups of countries or regions based on excess death rates, defined as excess death counts per 100,000 inhabitants, are considerably less informative than previously thought and should therefore be used with caution when evaluating different policies across countries. Compared to the standard WHO methodology, our frequentist model increases the average number of statistically indistinguishable units at a given rank from 6.2 to 27.1 among the 30 European countries in our database, and from 13.1 to 49.0 among the 51 U.S. jurisdictions

No abstract is available for this item.