MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning
Medical image segmentation is evolving from task-specific models toward generalizable frameworks. Recent research leverages Multi-modal Large Language Models (MLLMs) as autonomous agents, employing reinforcement learning with verifiable reward (RLVR) to orchestrate specialized tools like the Segment Anything Model (SAM). However, these approaches often rely on single-turn, rigid interaction strategies and lack process-level supervision during training, which hinders their ability to fully exploit the dynamic potential of interactive tools and leads to redundant actions. To bridge this gap, we propose MedSAM-Agent, a framework that reformulates interactive segmentation as a multi-step autonomous decision-making process. First, we introduce a hybrid prompting strategy for expert-curated trajectory generation, enabling the model to internalize human-like decision heuristics and adaptive refinement strategies. Furthermore, we develop a two-stage training pipeline that integrates multi-turn, end-to-end outcome verification with a clinical-fidelity process reward design to promote interaction parsimony and decision efficiency. Extensive experiments across 6 medical modalities and 21 datasets demonstrate that MedSAM-Agent achieves state-of-the-art performance, effectively unifying autonomous medical reasoning with robust, iterative optimization. Code is available https://github.com/CUHK-AIM-Group/MedSAM-Agent{here}.
6983428837bca54cd0587d0f
Shengyuan Liu
claimed_verified
2026-02-04T19:03:39.205Z
64745934a855203d8fe9588d
https://cdn-avatars.huggingface.co/v1/production/uploads/64745934a855203d8fe9588d/ETmY7zSULoAKrhShMvOzy.jpeg
Shengyuan Liu
Saint-lsy
user
Saint-lsy
6983428837bca54cd0587d10
Liuxin Bao
6983428837bca54cd0587d11
Qi Yang
6983428837bca54cd0587d12
Wanting Geng
6983428837bca54cd0587d13
Boyun Zheng
6983428837bca54cd0587d14
Chenxin Li
6983428837bca54cd0587d15
Wenting Chen
6983428837bca54cd0587d16
Houwen Peng
6983428837bca54cd0587d17
Yixuan Yuan
MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning
Medical image segmentation is evolving from task-specific models toward generalizable frameworks. Recent research leverages Multi-modal Large Language Models (MLLMs) as autonomous agents, employing reinforcement learning with verifiable reward (RLVR) to orchestrate specialized tools like the Segment Anything Model (SAM). However, these approaches often rely on single-turn, rigid interaction strategies and lack process-level supervision during training, which hinders their ability to fully exploit the dynamic potential of interactive tools and leads to redundant actions. To bridge this gap, we propose MedSAM-Agent, a framework that reformulates interactive segmentation as a multi-step autonomous decision-making process. First, we introduce a hybrid prompting strategy for expert-curated trajectory generation, enabling the model to internalize human-like decision heuristics and adaptive refinement strategies. Furthermore, we develop a two-stage training pipeline that integrates multi-turn, end-to-end outcome verification with a clinical-fidelity process reward design to promote interaction parsimony and decision efficiency. Extensive experiments across 6 medical modalities and 21 datasets demonstrate that MedSAM-Agent achieves state-of-the-art performance, effectively unifying autonomous medical reasoning with robust, iterative optimization. Code is available https://github.com/CUHK-AIM-Group/MedSAM-Agent{here}.
MedSAM-Agent reformulates medical image segmentation as a multi-step decision-making process using hybrid prompting and a two-stage training pipeline with process rewards to improve autonomous reasoning and optimization.
Multi-modal Large Language Models reinforcement learning with verifiable reward Segment Anything Model hybrid prompting strategy two-stage training pipeline clinical-fidelity process reward multi-turn interaction end-to-end outcome verification