Pervasive selection or is it.? why are FST outliers sometimes so frequent?

Abstract

It is now common for population geneticists to estimate FST for a large number of loci across the genome, before testing for selected loci as being outliers to the FST distribution. One surprising result of such FST scans is the often high proportion (>1% and sometimes >10%) of outliers detected, and this is often interpreted as evidence for pervasive local adaptation. In this issue of Molecular Ecolog, Fourcade et al. () observe that a particularly high rate of FST outliers has often been found in river organisms, such as fishes or damselflies, despite there being no obvious reason why selection should affect a larger proportion of the genomes of these organisms. Using computer simulations, Fourcade et al. () show that the strong correlation in co-ancestry produced in long one-dimensional landscapes (such as rivers, valleys, peninsulas, oceanic ridges or coastlines) greatly increases the neutral variance in FST, especially when the landscape is further reticulated into fractal networks. As a consequence, outlier tests have a high rate of false positives, unless this correlation can be taken into account. Fourcade et al.'s study highlights an extreme case of the general problem, first noticed by Robertson (,b) and Nei & Maruyama (), that correlated co-ancestry inflates the neutral variance in FST when compared to its expectation under an island model of population structure. Similar warnings about the validity of outlier tests have appeared regularly since then but have not been widely cited in the recent genomics literature. We further emphasize that FST outliers can arise in many different ways and that outlier tests are not designed for situations where the genetic architecture of local adaptation involves many loci. © 2013 Blackwell Publishing Ltd.