Optimal clustering of frequency data with application to disease risk categorization

We provide a clustering procedure for a special type of dataset, known as frequency data, which counts the frequency of a certain binary outcome. An interpretation of the data as a discrete distribution enables us to extract statistical information, which we embed within an optimization-based framework. Our analysis of the resulting combinatorial optimization problem allows us to reformulate it as a more tractable network flow problem. This, in turn, enables the construction of exact algorithms that converge to the optimal solution in quadratic time. In addition, to be able to handle large-scale datasets, we provide two hierarchical heuristic algorithms that run in linearithmic time. Our moment-based method results in clustering solutions that are shown to perform well for a family of applications. We illustrate the benefits of our findings through a case study on HIV risk categorization within the context of large-scale screening through group testing. Our results on CDC data show that the proposed clustering framework consistently outperforms other popular clustering methods.