Statistical framework for validation without ground truth of choroidal thickness changes detection

Abstract

Monitoring subtle choroidal thickness changes in the human eye delivers insight into the pathogenesis of various ocular diseases such as myopia and helps planning their treatment. However, a thorough evaluation of detection-performance is challenging as a ground truth for comparison is not available. Alternatively, an artificial ground truth can be generated by averaging the manual expert segmentations. This makes the ground truth very sensitive to ambiguities due to different interpretations by the experts. In order to circumvent this limitation, we present a novel validation approach that operates independently from a ground truth and is uniquely based on the common agreement between algorithm and experts. Utilizing an appropriate index, we compare the joint agreement of several raters with the algorithm and validate it against manual expert segmentation. To illustrate this, we conduct an observational study and evaluate the results obtained using our previously published registration-based method. In addition, we present an adapted state-of-the-art evaluation method, where a paired t-test is carried out after leaving out the results of one expert at the time. Automated and manual detection were performed on a dataset of 90 OCT 3D-volume stack pairs of healthy subjects between 8 and 18 years of age from Asian urban regions with a high prevalence of myopia.