Information Theoretic Criteria

The optimization criteria in information theoretic learning (ITL) have gained increasing attention over the past few years, which use the information theoretic quantities (e.g., entropy or correntropy) estimated directly from the data instead of the usual second-order statistical measures, such as variance and covariance, as the optimization costs. In ITL, the correntropy is a local similarity measure (only concerns about the local part of the error PDF falling within the kernel size), and it is naturally a robust cost, named maximum correntropy criterion (MCC). As another key principle in ITL, entropy measures the uncertainty of error; therefore, the optimization criterion is formulated as minimum error entropy (MEE) which aims to reduce the error uncertainty via minimizing Renyi’s quadratic entropy. Compared with MCC dealing with heavy-tailed non-Gaussian noises, the MEE improves the capability of model prediction when facing more complex non-Gaussian noises, such as noises from multimodal distributions. Sometimes, in order to obtain an optimal solution, the MEE needs to manually add a bias to the model to yield zero mean error. To more naturally adjust the error mean, the MEE with fiducial points (MEEF) was proposed, which can automatically anchor the error mean around zero. In this chapter, the MCC, MEE, and MEEF are briefly reviewed.