Maximum likelihood estimation of the heterogeneity of substitution rate among nucleotide sites

Abstract

This paper presents a maximum likelihood approach to estimating the variation of substitution rate among nucleotide sites. We assume that the rate varies among sites according to an invariant+gamma distribution, which has two parameters: the gamma parameter α and the proportion of invariable sites θ. Theoretical treatments on three, four, and five sequences have been conducted, and computer programs have been developed. It is shown that ρ = (1+θα)/(1+α) is a good measure for the rate heterogeneity among sites. Extensive simulations show that (1) if the proportion of invariable sites is negligible. i.e., θ = 0, the gamma parameter α can be satisfactorily estimated, even with three sequences; (2) if the proportion of invariable sites is not negligible, the heterogeneity ρ can still be suitably estimated with four or more sequences; and (3) the distances estimated by the proposed method are almost unbiased and are robust against violation of the assumption of the invariant+gamma distribution.