Determination of calculation parameters for difference approximation of group refractive index of air based on two-point central method

Abstract

Background: The accurate knowledge of the refractive index of air is very important for environmental compensation in length measurements. Methods: In this study, we apply difference approximation methods to facilitate the calculation of the group refractive index of air (GRA) over all calculable wavelengths. Our approach involves the determination of a suitable combination of the step size Δλ and numbers of significant digits of calculations of the phase refractive index of air (PRA) by balancing the two main errors (round-off and truncation errors) over the entire calculable wavelength range. Results: Based on our calculations, we find that the GRA computation over the range of all calculable wavelengths (301.5-1698.5 nm) can be easily approximated by the two-point central difference method with Δλ = 1.5 nm and 12-digit PRA accuracy. Conclusion: The approximation accuracy is less than 5 × 10-9. Our approach can be used by non-expert users to obtain the GRA with sufficient accuracy.