Conjugate Maximum Likelihood

In some important cases, maximum-likelihood estimation amounts to evaluating a conjugate, in which the data vector of sufficient statistics is the dual perturbation for the vector of parameters. A comparative-statics relationship applies. In a large sample, the variance of the parameter estimate is the derivative of the estimate with respect to the sufficient statistics. If the parameter estimate is sensitive to the data, then the variance of the estimate is large. The likelihood is the probability density, a function of the data and the parameters. In special but important cases, the log-likelihood can be expressed in the simple form (57.1) $$\left \langle \boldsymbol {x}^{\ast },\boldsymbol {x}\right \rangle -f\left ( \boldsymbol {x}\right ) -g\left ( \cdots \right ) ,$$ set in a Euclidean space. Here x is the vector of unknown parameters, to estimate by choosing x to maximize the likelihood. The vector x* is a data vector of sufficient statistics, paired with the parameter vector. The proper, closed, convex function f is a function of the parameters but is independent of the data. There can be an extra term g that depends on other data but is independent of the parameters. In this situation, maximum-likelihood estimation amounts to the evaluation of the conjugate (57.2) $$f^{\ast }\left ( \boldsymbol {x}^{\ast }\right ) =\sup _{\boldsymbol {x}}\left [ \left \langle \boldsymbol {x}^{\ast },\boldsymbol {x}\right \rangle -f\left ( \boldsymbol {x}\right ) \right ] ,$$ as the extra term has no effect on the estimate (The likelihood is not necessarily the probability density function of the sufficient statistic, even though one derives the likelihood from the probability density function of the full sample). A necessary condition for (57.1) is that the probability distribution belong to the natural exponential family Morris (1982). Morris emphasizes that the choice parameters may be a transformation of the traditional parameters employed by statisticians.