Epistatic Clustering: A Model-Based Approach for Identifying Links Between Clusters

Most clustering methods assume that the data can be represented by mutually exclusive clusters, although this assumption may not be the case in practice. For example, in gene expression microarray studies, investigators have often found that a gene can play multiple functions in a cell and may, therefore, belong to more than one cluster simultaneously, and that gene clusters can be linked to each other in certain pathways. This article examines the effect of the above assumption on the likelihood of finding latent clusters using theoretical calculations and simulation studies, for which the epistatic structures were known in advance, and on real data analyses. To explore potential links between clusters, we introduce an epistatic mixture model which extends the Gaussian mixture by including epistatic terms. A generalized expectation-maximization (EM) algorithm is developed to compute the related maximum likelihood estimators. The Bayesian information criterion is then used to determine the order of the proposed model. A bootstrap test is proposed for testing whether the epistatic mixture model is a significantly better fit to the data than a standard mixture model in which each data point belongs to one cluster. The asymptotic properties of the proposed estimators are also investigated when the number of analysis units is large. The results demonstrate that the epistatic links between clusters do have a serious effect on the accuracy of clustering and that our epistatic approach can substantially reduce such an effect and improve the fit.