SAM2S: Segment Anything in Surgical Videos via Semantic Long-term Tracking
Surgical video segmentation is crucial for computer-assisted surgery, enabling precise localization and tracking of instruments and tissues. Interactive Video Object Segmentation (iVOS) models such as Segment Anything Model 2 (SAM2) provide prompt-based flexibility beyond methods with predefined categories, but face challenges in surgical scenarios due to the domain gap and limited long-term tracking. To address these limitations, we construct SA-SV, the largest surgical iVOS benchmark with instance-level spatio-temporal annotations (masklets) spanning eight procedure types (61k frames, 1.6k masklets), enabling comprehensive development and evaluation for long-term tracking and zero-shot generalization. Building on SA-SV, we propose SAM2S, a foundation model enhancing SAM2 for Surgical iVOS through: (1) DiveMem, a trainable diverse memory mechanism for robust long-term tracking; (2) temporal semantic learning for instrument understanding; and (3) ambiguity-resilient learning to mitigate annotation inconsistencies across multi-source datasets. Extensive experiments demonstrate that fine-tuning on SA-SV enables substantial performance gains, with SAM2 improving by 12.99 average J\&F over vanilla SAM2. SAM2S further advances performance to 80.42 average J\&F, surpassing vanilla and fine-tuned SAM2 by 17.10 and 4.11 points respectively, while maintaining 68 FPS real-time inference and strong zero-shot generalization. Code and dataset will be released at https://jinlab-imvr.github.io/SAM2S.
691fd2cdcce0eb9b6387af75
Haofeng Liu
claimed_verified
2025-11-21T08:59:51.125Z
66bf75432777c05070bf49dc
/avatars/e18c2f6f5b1c37873dc181de4f255a8b.svg
Haofeng Liu
HeverLaw
user
HeverLaw
691fd2cdcce0eb9b6387af76
Ziyue Wang
691fd2cdcce0eb9b6387af77
Sudhanshu Mishra
691fd2cdcce0eb9b6387af78
Mingqi Gao
691fd2cdcce0eb9b6387af79
Guanyi Qin
691fd2cdcce0eb9b6387af7a
Chang Han Low
691fd2cdcce0eb9b6387af7b
Alex Y. W. Kong
claimed_verified
2025-11-21T08:59:47.726Z
691fd8847620f914ecec559f
/avatars/0d1688c65fbcb77de749ef0ff7e43eba.svg
Alex Kong
Awsdude
user
Awsdude
691fd2cdcce0eb9b6387af7c
Yueming Jin
SAM2S: Segment Anything in Surgical Videos via Semantic Long-term Tracking
Surgical video segmentation is crucial for computer-assisted surgery, enabling precise localization and tracking of instruments and tissues. Interactive Video Object Segmentation (iVOS) models such as Segment Anything Model 2 (SAM2) provide prompt-based flexibility beyond methods with predefined categories, but face challenges in surgical scenarios due to the domain gap and limited long-term tracking. To address these limitations, we construct SA-SV, the largest surgical iVOS benchmark with instance-level spatio-temporal annotations (masklets) spanning eight procedure types (61k frames, 1.6k masklets), enabling comprehensive development and evaluation for long-term tracking and zero-shot generalization. Building on SA-SV, we propose SAM2S, a foundation model enhancing SAM2 for Surgical iVOS through: (1) DiveMem, a trainable diverse memory mechanism for robust long-term tracking; (2) temporal semantic learning for instrument understanding; and (3) ambiguity-resilient learning to mitigate annotation inconsistencies across multi-source datasets. Extensive experiments demonstrate that fine-tuning on SA-SV enables substantial performance gains, with SAM2 improving by 12.99 average J\&F over vanilla SAM2. SAM2S further advances performance to 80.42 average J\&F, surpassing vanilla and fine-tuned SAM2 by 17.10 and 4.11 points respectively, while maintaining 68 FPS real-time inference and strong zero-shot generalization. Code and dataset will be released at https://jinlab-imvr.github.io/SAM2S.
A surgical video segmentation model SAM2S enhances interactive video object segmentation through robust memory, temporal learning, and ambiguity handling, achieving high performance and real-time inference on a comprehensive surgical benchmark.
Interactive Video Object Segmentation iVOS Segment Anything Model 2 SAM2 SA-SV DiveMem temporal semantic learning ambiguity-resilient learning $\mathcal{J}$\&$\mathcal{F}$ real-time inference zero-shot generalization