MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.
699e64a9dfbcf0b800aecaa9
Taha Koleilat
claimed_verified
2026-02-27T16:44:13.587Z
67d33a3b1d5d1fd92c3cfce4
https://cdn-avatars.huggingface.co/v1/production/uploads/67d33a3b1d5d1fd92c3cfce4/1f-hKaMMPBgMUbVXy3an-.jpeg
Taha Koleilat
TahaKoleilat
user
TahaKoleilat
699e64a9dfbcf0b800aecaaa
Hojat Asgariandehkordi
699e64a9dfbcf0b800aecaab
Omid Nejati Manzari
699e64a9dfbcf0b800aecaac
Berardino Barile
699e64a9dfbcf0b800aecaad
Yiming Xiao
699e64a9dfbcf0b800aecaae
Hassan Rivaz
MedCLIPSeg: Probabilistic Vision-Language Adaptation for Data-Efficient and Generalizable Medical Image Segmentation
Medical image segmentation remains challenging due to limited annotations for training, ambiguous anatomical features, and domain shifts. While vision-language models such as CLIP offer strong cross-modal representations, their potential for dense, text-guided medical image segmentation remains underexplored. We present MedCLIPSeg, a novel framework that adapts CLIP for robust, data-efficient, and uncertainty-aware medical image segmentation. Our approach leverages patch-level CLIP embeddings through probabilistic cross-modal attention, enabling bidirectional interaction between image and text tokens and explicit modeling of predictive uncertainty. Together with a soft patch-level contrastive loss that encourages more nuanced semantic learning across diverse textual prompts, MedCLIPSeg effectively improves data efficiency and domain generalizability. Extensive experiments across 16 datasets spanning five imaging modalities and six organs demonstrate that MedCLIPSeg outperforms prior methods in accuracy, efficiency, and robustness, while providing interpretable uncertainty maps that highlight local reliability of segmentation results. This work demonstrates the potential of probabilistic vision-language modeling for text-driven medical image segmentation.
MedCLIPSeg adapts CLIP for medical image segmentation by leveraging patch-level embeddings and probabilistic attention to achieve data-efficient, uncertain-aware segmentation with interpretability.
vision-language models CLIP medical image segmentation probabilistic cross-modal attention patch-level embeddings soft patch-level contrastive loss uncertainty-aware segmentation domain generalizability