A Labeling Intercomparison of Retrogressive Thaw Slumps by a Diverse Group of Domain Experts

Abstract

Deep-learning (DL) models have become increasingly beneficial for the detection of retrogressive thaw slumps (RTS) in the permafrost domain. However, comparing accuracy metrics is challenging due to unstandardized labeling guidelines. To address this, we conducted an experiment with 12 international domain experts from a broad range of scientific backgrounds. Using 3 m PlanetScope multispectral imagery, they digitized RTS footprints in two sites. We evaluated label uncertainty by comparing manually outlined RTS labels using Intersection-over-Union (IoU) and F1 metrics. At the Canadian Peel Plateau site, we see good agreement, particularly in the active parts of RTS. Differences were observed in the interpretation of the debris tongue and the stable vegetated sections of RTS. At the Russian Bykovsky site, we observed a larger mismatch. Here, the same differences were documented, but several participants mistakenly identified non-RTS features. This emphasizes the importance of site-specific knowledge for reliable label creation. The experiment highlights the need for standardized labeling procedures and definition of their scientific purpose. The most similar expert labels outperformed the accuracy metrics reported in the literature, highlighting human labeling capabilities with proper training, site knowledge, and clear guidelines. These findings lay the groundwork for DL-based RTS monitoring in the pan-Arctic.