Enhanced YOLO11n-Seg with Attention Mechanism and Geometric Metric Optimization for Instance Segmentation of Ripe Blueberries in Complex Greenhouse Environments

This study proposes an improved YOLO11n-seg instance segmentation model to address the limitations of existing models in accurately identifying mature blueberries in complex greenhouse environments. Current methods often lack sufficient accuracy when dealing with complex scenarios, such as fruit occlusion, lighting variations, and target overlap. To overcome these challenges, we developed a novel approach that integrates a Spatial–Channel Adaptive (SCA) attention mechanism and a Dual Attention Balancing (DAB) module. The SCA mechanism dynamically adjusts the receptive field through deformable convolutions and fuses multi-scale color features. This enhances the model’s ability to recognize occluded targets and improves its adaptability to variations in lighting. The DAB module combines channel–spatial attention and structural reparameterization techniques. This optimizes the YOLO11n structure and effectively suppresses background interference. Consequently, the model’s accuracy in recognizing fruit contours improves. Additionally, we introduce Normalized Wasserstein Distance (NWD) to replace the traditional intersection over union (IoU) metric and address bias issues that arise in dense small object matching. Experimental results demonstrate that the improved model significantly improves target detection accuracy, recall rate, and mAP@0.5, achieving increases of 1.8%, 1.5%, and 0.5%, respectively, over the baseline model. On our self-built greenhouse blueberry dataset, the mask segmentation accuracy, recall rate, and mAP@0.5 increased by 0.8%, 1.2%, and 0.1%, respectively. In tests across six complex scenarios, the improved model demonstrated greater robustness than mainstream models such as YOLOv8n-seg, YOLOv8n-seg-p6, and YOLOv9c-seg, especially in scenes with dense occlusions. The improvement in mAP@0.5 and F1 scores validates the effectiveness of combining attention mechanisms and multiple metric optimizations, for instance, segmentation tasks in complex agricultural scenes.