Beta-coalescents when sample size is large

TL;DR

The authors extend sweepstakes-reproduction models by imposing an upper bound on offspring, connecting the bound’s growth relative to population size to convergence toward Kingman, incomplete Beta, or complete Beta coalescents. They derive precise timescale and error bounds ($c_N$ and related terms) and identify three distinct asymptotic regimes for the coalescent limit depending on how the bound scales with $N$. Through theory and simulations, they show that sample size can drive breakdowns of the coalescent approximation in nuanced ways, and that the site frequency spectrum may still reflect the limiting Beta-tree behavior even when the full genealogical tree deviates. Conditioning on population ancestry can modestly affect predictions, while mis-specifying the bound can bias inferences about $oldsymbol{oldsymbol{ extalpha}}$ and the coalescent type. Overall, the work provides a nuanced framework for modeling highly skewed offspring distributions in large samples and clarifies when different coalescent limits apply under ecologically realistic bounds.

Abstract

Sweepstakes reproduction refers to a highly skewed individual recruitment success without involving natural selection and may apply to individuals in broadcast spawning populations characterised by Type III survivorship. We consider an extension of the model of sweepstakes reproduction for a haploid panmictic population of constant size $N$; the extension also works as an alternative to the Wright-Fisher model. Our model incorporates an upper bound on the random number of potential offspring (juveniles) produced by a given individual. Depending on how the bound behaves relative to the total population size, we obtain the Kingman coalescent, an incomplete Beta-coalescent, or the (complete) Beta-coalescent. We argue that applying such an upper bound is biologically reasonable. Moreover, we estimate the error of the coalescent approximation. The error estimates reveal that convergence can be slow, and small sample size can be sufficient to invalidate convergence, for example if the stated bound is of the form $N/\log N$. We use simulations to investigate the effect of increasing sample size on the site-frequency spectrum. When the limit is a Beta-coalescent, the site frequency spectrum will be as predicted by the limiting tree even though the full coalescent tree may deviate from the limiting one. When in the domain of attraction of the Kingman coalescent the effect of increasing sample size depends on the effective population size as has been noted in the case of the Wright-Fisher model. Conditioning on the population ancestry (the random ancestral relations of the entire population at all times) may have little effect on the site-frequency spectrum for the models considered here (as evidenced by simulation results).

Figures (8)

• Figure 1: Two trees for $n=4$, mutations are shown as '$\circ$'; the observed site-frequency spectra would not allow to distinguish between the two gene genealogies
• Figure 2: Illustration of events leading to different tree topologies
• Figure 2: Numerical density estimates of the distribution of the tree size of the population tree of a haploid population of constant size $N$ (as shown) evolving according to the Wright-Fisher model. See Appendix \ref{['sec:tree-size-wright']} for the code.
• Figure 3: Comparing relative branch lengths -- the Kingman coalescent. Comparison of $\overline \varrho_{i}^{N}(n)$ (symbols) and of $\overline \varrho_{i}(n)$ (black lines) shown as $\log(e_{i}(n)) - \log(1 -e_{i}(n))$ as a function of $\log(i/n) - \log(1-i/n)$ where $e_{i}(n)$ denotes the corresponding mean relative branch length estimate for sample size $n$ as shown from a haploid panmictic population of constant size $N = 10^{4}$ evolving according to Definition \ref{['Schwm']} with number of potential offspring distributed according to \ref{['eq:haploid_pxi']} with $\alpha$ and $\psi(N)$ as shown. The black lines are the approximation $\mathds{E}\left[ B_{i}(n) \right]/\mathds{E}\left[ B(n) \right] = i^{-1}/\sum_{j=1}^{n-1}j^{-1}$ of $\mathds{E}\left[ R_{i}(n) \right]$ predicted by the Kingman coalescent F95. The cyan lines for the case $n = N$ is a loess regression (using the function loess in Rrsystem) through $\overline \varrho_{i}^{N}(n)$ for $i\in \{2,3, \ldots, n-1\}$; the estimates of $\mathds{E}\left[ R_{i}^{N}(n) \right]$ are the results from $10^{4}$ experiments. Appendix \ref{['sec:code']} contains a brief description of an algorithm for estimating $\mathds{E}\left[ R_{i}^{N}(n) \right]$
• Figure 4: Comparing relative branch lengths - Beta-coalescents. Comparison of $\overline \varrho_i^N(n)$ ( symbols) and of $\overline\varrho_i(n)$ (lines) shown as $\log(e_{i}(n)) - \log(1 -e_{i}(n))$ as a function of $\log(i/n) - \log(1-i/n)$, where $e_{i}(n)$ denotes the corresponding mean relative branch length estimate for sample size $n$ as shown from a haploid population of constant size $N = 10^{4}$ evolving according to Definition \ref{['Schwm']} with number of potential offspring distributed according to \ref{['eq:haploid_pxi']} with $\alpha=1.05$ and $\psi(N) = N$ (a), and according to \ref{['eq:26']} with $\alpha = 1.05$ (b). The blue dashed lines show $\overline \varrho_i(n)$ for the complete Beta-coalescent, and the red solid lines in (a) show $\overline \varrho_{i}(n)$ for the incomplete Beta-coalescent with $K=1$. The cyan lines for the case $n = N$ is a loess regression (using the function loess in Rrsystem) through $\overline\varrho_{i}^{N}(n)$ for $i\in \{2,3, \ldots, n-1\}$; the estimates of $\mathds{E}\left[ R_{i}^{N}(n) \right]$ are the results from $10^{4}$ experiments. Appendix \ref{['sec:code']} contains a brief description of an algorithm for estimating $\mathds{E}\left[ R_{i}^{N}(n) \right]$

Theorems & Definitions (46)

• Definition 2.1: Notation
• Definition 2.2: The (complete) Beta coalescent schweinsberg03
• Definition 2.3: The Schweinsberg model schweinsberg03
• Definition 2.4: The coalescence probability $c_{N}$
• Definition 2.5: Effective population size
• Theorem 2.6: schweinsberg03
• Remark 3.1
• Proposition 3.2
• Proposition 3.3
• Proposition 3.4