Impact of Attention Mechanisms and Focal Loss Tuning on RetinaNet Performance for Crop-Weed Detection: A Comparative Study with Anchor-Free Detectors

Abstract

Weed control remains a major challenge in agricultural production, as weeds compete with crops for essential resources such as water, nutrients, and sunlight, which reduces yield and sustainability. Advances in deep learning and computer vision have facilitated the development of automated weed detection systems that function effectively in complex field environments. This study conducts a systematic ablation analysis of attention mechanisms and focal loss optimization within the RetinaNet architecture for tobacco and weed detection in small-scale agricultural image datasets. Six model variants are assessed: a vanilla RetinaNet baseline, focal loss optimization alone, and three lightweight attention modules (CBAM, ECA, and Coordinate Attention), each combined with optimized focal loss parameters (α =0.3, γ=2.5). To benchmark performance against contemporary detectors, YOLOv11n is included as an anchor-free baseline. Experiments utilize a dataset of 307 field images with a 70/15/15 train/validation/test split to ensure independent evaluation. Results indicate that ECA combined with focal loss achieves the highest mAP@50 (0.760) and mAP@75 (0.477) among RetinaNet variants, representing an 8.3% improvement in localization quality over the vanilla baseline. Coordinate Attention attains the highest recall (0.611), highlighting its suitability for detection-critical applications. Ablation results also demonstrate that CBAM without focal loss substantially reduces performance, emphasizing the importance of loss stabilization in small-dataset scenarios. Although YOLOv11n achieves a higher overall mAP (0.511 vs. 0.442), the proposed attention-enhanced RetinaNet variants deliver competitive mAP@50 performance with interpretable, modular enhancements. These findings offer practical guidance for designing attention-integrated detectors in precision agriculture applications with limited annotation resources.