Coancestry superposed on admixed populations yields measures of relatedness at individual-level resolution

The admixture model is widely applied to estimate and interpret population structure among individuals. Here we consider a “standard admixture” model that assumes the admixed populations are unrelated and also a generalized model, where the admixed populations themselves are related via coancestry (or covariance) of allele frequencies. The generalized model yields a potentially more realistic and substantially more flexible model that we call “super admixture”. This super admixture model provides a one-to-one mapping in terms of probability moments with the kinship model, the latter of which is a general model of genome-wide relatedness and structure based on identity-by-descent. We introduce a method to estimate the super admixture model that is based on method of moments, does not rely on likelihoods, is computationally efficient, and scales to massive sample sizes. We apply the method to several human data sets and show that the admixed populations are indeed substantially related, implying the proposed method captures a new and important component of evolutionary history and structure in the admixture model. We show that the fitted super admixture model estimates relatedness between all pairs of individuals at a resolution similar to the kinship model. The super admixture model therefore provides a tractable, forward-generating probabilistic model of complex structure and relatedness that should be useful in a variety of scenarios.Author summary: Characterizing patterns of genetic relatedness and structure among individuals is a central topic in genetics. A widely used framework for this is the admixture model, which views each individual’s genome as a mosaic derived from several antecedent populations. However, a standard admixture model assumes the antecedent populations are unrelated, an assumption that may not adequately capture the complexity of human population structure. In this work, we consider an extended model to allow the admixed antecedent populations themselves to be related to each other, a framework we call “super admixture”. We introduce algorithms for estimating the super admixture model and simulating genotypes under this model that are likelihood-free, computationally efficient, and scalable. Applying our approach to several human genetic data sets, we show that the super admixture model uncovers finer structure among the admixed populations, and thereby provides measures of relatedness at individual-level resolution.