Information and Entropy in Cluster Analysis

Cluster analysis provides methods for subdividing a set of objects into a suitable number of ‘classes’, ‘groups’, or ‘types’ C 1,…,C m such that each class is as homogeneous as possible and different classes are sufficiently separated. This paper shows how entropy and information measures have been or can be used in this framework. We present several probabilistic clustering approaches which are related to, or lead to, information and entropy criteria g(C) for selecting an optimum partition C = (C 1,…,C m ) of n data vectors, for qualitative and for quantitative data, assuming loglinear, logistic, and normal distribution models, together with appropriate iterative clustering algorithms. A new partitioning problem is considered in Section 5 where we look for a dissection (discretization) C of an arbitrary sample space Y (e.g. R p or 0,1 p ) such that the ø—divergence I c (P 0, P 1) between two discretized distributions P o (C i ), P 1(C i ) (i = 1,…, m) will be maximized (e.g., Kullback-Leibler’s discrimination information or the X 2 noncentrality parameter). We conclude with some comments on methods for selecting a suitable number of classes, e.g., by using Akaike’s information criterion AIC and its modifications.