Understanding Novice's Annotation Process For 3D Semantic Segmentation Task With Human-In-The-Loop

Abstract

Large-scale 3D point clouds are often used as training data for 3D semantic segmentation, but the labor-intensive nature of the annotation process challenges the acquisition of sufficient labeled data. Meanwhile, there has been limited research on introducing novice annotators to acquire the labeled data by enhancing their annotation performance and user experience. Therefore, in this study, we explored solutions involving two dimensions: the presence of AI assistance and the number of classes visualized simultaneously in model's segmentation results in HITL. We conducted a user study with 16 novice annotators who had no prior experience in 3D semantic segmentation, asking them to perform annotation tasks. The results revealed an interaction effect between the two dimensions on annotation accuracy and labeling efficiency. We also found that displaying multiple classes at once reduced the time taken for annotation. Moreover, visualizing multiple classes at once or the absence of AI assistance led to a greater increase in model accuracy compared to our baselines. The best user experience was observed when the visualization showed a single class at a time with AI assistance. Based on these findings, we discuss which environments can enhance novice annotators' annotation performance and user experience in 3D semantic segmentation tasks within HITL contexts.