How to Measure the Error of an M-Estimate and Report It Conservatively

We construct a conservative error reporting function for the accuracy of M-estimators of a multi-dimensional statistical parameter in the context of a fixed finite number of independent but not identically distributed observations. The requirement that error estimates be conservative has been proposed by Kiefer, Brown, and Berger, but it has not previously been studied in a general context where exact calculations are impossible. In the case of maximum likelihood estimation our construction leads to an error-reporting function which dominates the likelihood ratio statistic. We specialize further with a variety of generalized linear model examples where we measure error using data-dependent loss functions dominated by the likelihood ratio statistic. We demonstrate analogous results for mixture models and least pth power regression.