Evolution under Stochastic Transmission: Mutation-Rate Modifiers

TL;DR

The analysis shows how stochasticity and recombination in transmission do not simply modify the magnitude of evolutionary change predicted under deterministic assumptions, but can generate conditions under which the direction of modifier evolution is qualitatively reversed relative to the deterministic Reduction Principle.

Abstract

Evolutionary analyses of large populations commonly incorporate stochasticity through temporal variation in selection while treating genetic transmission as fixed. Much less attention has been given to stochasticity in transmission itself. We study a selected locus with alleles $A$ and $a$ under constant selection, linked to a neutral modifier locus whose alleles $M_1$ and $M_2$ control the mutation rate from $A$ to $a$. Under constant transmission, the Reduction Principle applies: near a mutation--selection balance where $M_1$ is fixed with mutation rate $u_1$, a rare allele $M_2$ invades if its associated rate $u_2$ is smaller than $u_1$, but cannot invade if $u_2$ is larger than $u_1$. This result holds for both haploid and diploid populations and is independent of recombination, which affects only the rate, not the direction, of evolutionary change. We extend this framework by allowing the mutation rate associated with the invading modifier to fluctuate randomly across generations. In this stochastic setting, invasion is no longer determined by mean mutation rates alone. Instead, it depends on the temporal distribution of mutation rates, the strength of selection at the selected locus, and the recombination rate between modifier and target. Stochastic transmission and recombination therefore do not merely rescale deterministic predictions based on the Reduction Principle; they can alter the direction of selection on modifier alleles.

Figures (4)

• Figure 1: Distributions on $[0,1]$ with a common mean $u_2 = 0.048796$. Points show simulated realizations of $u_{2,t}$ and solid curves show kernel density estimates. Panels correspond to uniform, beta, truncated gamma, and truncated log–normal distributions. In each case, parameters are chosen to produce the largest variance attainable within the given family under the constraint of support on $[0,1]$ and fixed mean (up to numerical precision). Reported variance values illustrate the wide disparity in admissible temporal variability across distributions despite identical means and bounds.
• Figure 2: Fixation probability of a modifier allele under temporally fluctuating mutation rates. Panels correspond to the distribution governing the across-generation mutation rate $u_{2,t}$ induced by $M_2$ (uniform, beta, truncated log-normal, truncated gamma). Axes show $\mathrm{Var}(u_{2,t})$ and $\mathbb{E}[u_{2,t}]$. Color denotes the final density of the modifier allele $M_2$ at generation $5000$, averaged across replicate simulations. The black contour marks the $50\%$ fixation boundary. All simulations assume complete linkage ($R=0$), selection coefficient $s=0.2$, nd baseline mutation rate $u_1=0.048796$.
• Figure 3: Top Lyapunov growth rate $\gamma(R)$ as a function of recombination rate $R$. For haploid (left column) and diploid (right column) populations under stochastic mutation. Mutation rates $u_{2,t}$ are i.i.d. Beta-distributed with fixed mean $\mathbb{E}[u_{2,t}] = 0.04$ and varying variance $\operatorname{Var}(u_2)$; the deterministic case corresponds to $\operatorname{Var}=0$. Curves show the top Lyapunov exponent of the random matrix product. Blue and red shaded regions indicate $\gamma(R)<0$ and $\gamma(R)>0$, respectively. Vertical dotted lines mark $R=0$ and $R=1/2$. Selection coefficients are $s=0.06$ (top row) and $s=0.20$ (bottom row).
• Figure 4: Fixation outcomes versus Lyapunov thresholds. Each panel corresponds to one mutation–rate distribution. Settings: $5{,}000$ generations, $R=0$, $s=0.2$, $u_1\approx0.048796$. Colors indicate final $M_2$ density. The red curve shows the Lyapunov prediction $\gamma(u_2^\ast)=0$. Across all distributions, the predicted threshold closely tracks the observed transition between invasion and loss.