UAMP: Consistent video object segmentation with uncertainty-aware memory propagation

The Segment Anything Model 2 (SAM 2) has emerged as a robust foundational model for video object segmentation. Nevertheless, it exhibits notable limitations in crowded scenarios, particularly those involving fast-moving objects or self-occlusion. Furthermore, its greedy-selection memory architecture is afflicted by “error accumulation,” which collectively impairs its performance in consistent object segmentation tasks. To address these critical drawbacks and enhance the consistency and robustness of SAM 2 in complex video scenarios, an enhanced variant of SAM 2, termed UAMP, is proposed, which integrates memory propagation based on explicit appearance and motion uncertainty modeling, coupled with a dual mechanism encompassing long-term memory updating and short-term memory selection. By fusing uncertainty-aware representations with these dual memory mechanisms, UAMP effectively accommodates dynamic variations in object appearance and motion, further refines the object memory bank, and thereby realizes consistent video object segmentation. Quantitative and qualitative evaluations conducted on diverse benchmark datasets demonstrate that UAMP achieves superior performance, particularly in scenarios involving occlusions and object reappearances. Specifically, UAMP yields consistent performance improvements over state-of-the-art (SOTA) methods across five video object segmentation (VOS) benchmarks, with a maximum enhancement of 5.6 points in the J&F metric. These findings underscore the robustness and effectiveness of the proposed UAMP method in addressing complex tracking scenarios, thus providing a valuable enhancement to SAM 2 for practical video object segmentation applications.