On Optimal Correlation-Based Prediction

This note examines, at the population-level, the approach of obtaining predictors h˜(X) of a random variable Y, given the joint distribution of (Y,X), by maximizing the mapping h↦κ(Y,h(X)) for a given correlation function κ(·,·). Commencing with Pearson’s correlation function, the class of such predictors is uncountably infinite. The least-squares predictor h* is an element of this class obtained by equating the expectations of Y and h(X) to be equal and the variances of h(X) and E(Y|X) to be also equal. On the other hand, replacing the second condition by the equality of the variances of Y and h(X), a natural requirement for some calibration problems, the unique predictor h** that is obtained has the maximum value of Lin’s (1989) concordance correlation coefficient (CCC) with Y among all predictors. Since the CCC measures the degree of agreement, the new predictor h** is called the maximal agreement predictor. These predictors are illustrated for three special distributions: the multivariate normal distribution; the exponential distribution, conditional on covariates; and the Dirichlet distribution. The exponential distribution is relevant in survival analysis or in reliability settings, while the Dirichlet distribution is relevant for compositional data.