Maximum Likelihood Method

A classical problem in the statistical decision theory is to estimate the probability distribution of a random vector X given its independent observations $$x_{1},\ldots,x_{n}$$ . Often it is assumed that the probability distribution comes from some family of functions parametrized by a set of parameters $$\theta _{1},\ldots,\theta _{m}$$ , so that in this case, the problem is reduced to estimating $$\theta _{1},\ldots,\theta _{m}$$ and is called parametric estimation. However, if no specific family of distributions is assumed, i.e., the probability distribution can not be completely defined by a finite number of parameters, the problem is called nonparametric estimation. In both parametric and nonparametric estimations, there are several approaches to determine the probability distribution in question: the maximum likelihood principle, maximum entropy principle, and the minimum relative entropy principle (or the principle of minimum discrimination information). These principles are closely related and are the subject of this chapter and the next one.