Comparison between total least squares and ordinary least squares in obtaining the linear relationship between stable water isotopes

Abstract

The linear relationship between two stable water isotopes (δD and δ18O) has been used to examine the physical processes and movements or changes of three water phases (water vapor, liquid water and ice), including deuterium excess. The ordinary least squares (OLS) method has been the most commonly used method to fit the linear relationship between two isotopic compositions of water. However, an alternative method, the total least squares (TLS) method, has been proposed because it considers the presence of errors in the explanatory variable (horizontal axis, δ18O). However, not many studies have examined the differences of the relationship using two stable isotopes between the OLS and TLS for various types of water. In this work, these two methods were compared using isotopic compositions of three types of water (Antarctic snow, water vapor and summer and winter rainfall). Statistically, the slopes and intercepts obtained by the two linear regression methods were not significantly different except for summer rainfall, which has the smallest coefficient of variations (R2). The TLS method produced larger slopes than the OLS method and the degrees of difference between the two methods were greater when the coefficient of variation was lower. In addition, with a Monte Carlo method, we showed that the differences between the two methods increased as the uncertainty increased. Moreover, the results of Bayesian linear regression were consistent with the two linear regressions. Although the TLS method is theoretically more suited to the linear regression for the stable water isotopes than the OLS method is, the application of the widely used OLS method can be recommended in the case of small measurements uncertainties after testing whether the linear parameters, slopes and intercepts, derived from the two methods are statistically significant different.