The Yule-$Λ$ Nested Coalescent: Distribution of the Number of Lineages
Toni Gui
TL;DR
The paper analyzes the Yule-$\Lambda$ nested coalescent, combining a Yule species-branching process with within-species $\Lambda$-coalescence. It introduces the fixed-point operator $G_c$ and proves the existence of a finite-mean fixed point $\nu_c^*$ in $\mathcal{S}_1$ when $0<c<E[1/X]$, along with convergence in distribution of the counts of lineages per surviving species to $\nu_c^*$. It further shows that for some $c$ there exists an additional fixed point $\nu_c'$ in $\mathcal{S}$ which may have infinite mean, with the mass at infinity depending on $P(X=1)$. The results rely on recursive distributional equations, generating function techniques, and delicate probabilistic couplings, highlighting both uniqueness in the finite-mean regime and non-uniqueness when infinite mean is possible. These findings contribute to understanding multi-species genealogies under Λ-coalescence, revealing how inter-species dynamics affect the distribution of intra-species lineages and the role of dust in fixed-point behavior.
Abstract
We study a model of a population with individuals sampled from different species. The Yule-$Λ$ nested coalescent describes the genealogy of the sample when each species merges with another randomly chosen species with a constant rate $c$ and the mergers of individuals in each species follow the $Λ$-coalescent. For the Yule-$Λ$ nested coalescent with $c<\int_0^1x^{-1}Λ(dx)<\infty$, where $Λ$ is the measure that characterizes the $Λ$-coalescent, we show that under some initial conditions, the distribution of the number of individual lineages belonging to one species converges weakly to the distribution $ν_c^*$, which is the solution to some recursive distributional equation (RDE) with finite mean. In addition, we show that for some values of $c$, the RDE has another solution with infinite mean.
