An Extension of the Kullback–Leibler Divergence for the Application of Machine Learning Classification Algorithms to Hierarchical Cluster Analysis

This paper demonstrates that the Kullback–Leibler (KL) divergence expression can be used to calculate distances between sample centroids when the probability distributions are replaced by classification probabilities. To test the properties of the KL distances, we used a large number of real and artificial datasets and many classification methods, the most important of which are as follows: linear discriminant analysis (LDA), flexible discriminant analysis (FDA), linear discriminant analysis using the generalized singular value decomposition (GSVD), regularized discriminant analysis (RDA), multinomial logistic regression (MLR), mixture discriminant analysis (MDA), naïve Bayes classification (NBC), artificial neural networks (ANN), support vector machines (SVM), and k-nearest neighbors (kNN). It was found that in general, there is a high correlation between Mahalanobis and KL distances exhibiting typical Mahalanobis-like properties, especially when using LDA, FDA, GSVD, RDA, and in many cases MDA. A Euclidean-like KL distance arises only from the NBC classification method in continuous datasets. The remaining KL distances, except for kNN in many datasets, tend to give clusters similar to those of MD, but significant deviations from MD can also be observed. The KL distances obtained mainly from kNN show serious problems when used in hierarchical cluster analysis. Finally, the majority of the classification methods examined in this paper can handle all types of data, continuous, ordinal, categorical, and binary, and therefore the corresponding KL distances can be calculated using any type of data or combination of data.