Rounding error, an unexpected fault in the output from a recording spectrophotometer: Implications for model discrimination

Abstract

Although commonly ignored in discussions of experimental error, rounding may sometimes be the major source of error, especially with modern precision instruments: some recording spectrophotometers are optically and photometrically capable of making absorbance measurements with errors less than 0.0003, but provide no numerical information more precise than ± 0.001. The problem may be diagnosed by a characteristic arrangement of points in a residual plot, which resembles the result of cutting a stroboscopic picture of a bouncing ball into several strips and modifying it by sliding the strips relative to one another to bring the points closer to the axis. Harmful effects of rounding error can be critical in experiments designed for model discrimination.