Asymmetric Distribution of Gene Trees Can Arise under Purifying Selection if Differences in Population Size Exist

Abstract

Incomplete lineage sorting (ILS) is an important factor that causes gene tree discordance. For gene trees of three species, under neutrality, random mating, and the absence of interspecific gene flow, ILS creates a symmetric distribution of gene trees: the gene tree that accords with the species tree has the highest frequency, and the two discordant trees are equally frequent. If the neutral condition is violated, the impact of ILS may change, altering the gene tree distribution. Here, we show that under purifying selection, even assuming that the fitness effect of mutations is constant throughout the species tree, if differences in population size exist among species, asymmetric distributions of gene trees will arise, which is different from the expectation under neutrality. In extremes, one of the discordant trees rather than the concordant tree becomes the most frequent gene tree. In addition, we found that in a real case, the position of Scandentia relative to Primate and Glires, the symmetry in the gene tree distribution can be influenced by the strength of purifying selection. In current phylogenetic inference, the impact of purifying selection on the gene tree distribution is rarely considered by researchers. This study highlights the necessity of considering this impact.