The coalescent in two partially isolated diffusion populations

Abstract

The n0 coalescent of Kingman (1982a, b) describes the family relationships among a sample of n0 individuals drawn from a panmictic species. It is a stochastic process resulting from n0 − 1 independent random events (coalescences) at each of which n (2 ≤ n ≤ n0) ancestral lineages of a sample are descended from n − 1 distinct ancestors for the first time. Here a similar genealogical process is studied for a species consisting of two populations with migration between them. The main interest is with the probability density of the time length between two successive coalescences and the spatial distribution of n − 1 ancestral lineages over two populations when n to n − 1 coalescence takes place. These are formulated based on a non-linear birth and death process with killing, and are used to derive several explicit formulae in selectively neutral population genetics models. To confirm and supplement the analytical results, a simulation method is proposed based on the underlying bivariate Markov chain. This method provides a general way for solving the present problem even when an analytical approach appears very difficult. It becomes clear that the effects of the present population structure are most conspicuous on 2 to 1 coalescence, with lesser extents on n to n − 1 (3 ≤ n) coalescence.This implies that in a more general model of population structure, the number of populations and the way in which a sample is drawn are important factors which determine the n0 coalescent. © 1988, Cambridge University Press. All rights reserved.