Evolution under Stochastic Transmission: Mutation-rate Modifiers

Evolutionary analyses of large populations commonly incorporate stochasticity through temporal variation in selection while treating genetic transmission parameters as fixed. Much less attention has been given to stochasticity in transmission itself. We study a selected locus with alleles A and a under constant selection, linked to a neutral modifier locus whose alleles M1 and M2 control the mutation rate from A to a. Under constant transmission, the Reduction Principle applies: near a mutation–selection balance where M1 is fixed with mutation rate u1, a rare allele M2 invades if its associated rate u2u1. This result holds for both haploid and diploid populations and is independent of other parameters such as recombination, which affects only the rate, not the direction, of change. We extend this framework by allowing the mutation rate associated with the invading modifier allele M2 to fluctuate randomly across generations. In this stochastic setting, invasion is no longer determined by mean mutation rates alone. Instead, it depends on the temporal distribution of mutation rates, the strength of selection at the focal locus, and the recombination rate between modifier and target loci. Stochastic transmission therefore does not merely rescale deterministic predictions based on the Reduction Principle; it can alter the direction of selection acting on modifier alleles.