Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset
The applicability of current lesion segmentation models for chest X-rays (CXRs) has been limited both by a small number of target labels and the reliance on long, detailed expert-level text inputs, creating a barrier to practical use. To address these limitations, we introduce a new paradigm: instruction-guided lesion segmentation (ILS), which is designed to segment diverse lesion types based on simple, user-friendly instructions. Under this paradigm, we construct MIMIC-ILS, the first large-scale instruction-answer dataset for CXR lesion segmentation, using our fully automated multimodal pipeline that generates annotations from chest X-ray images and their corresponding reports. MIMIC-ILS contains 1.1M instruction-answer pairs derived from 192K images and 91K unique segmentation masks, covering seven major lesion types. To empirically demonstrate its utility, we introduce ROSALIA, a vision-language model fine-tuned on MIMIC-ILS. ROSALIA can segment diverse lesions and provide textual explanations in response to user instructions. The model achieves high segmentation and textual accuracy in our newly proposed task, highlighting the effectiveness of our pipeline and the value of MIMIC-ILS as a foundational resource for pixel-level CXR lesion grounding.
691e88213c64d32b0364590c
Geon Choi
claimed_verified
2025-11-20T08:48:57.580Z
66714579562ee91c16be977c
/avatars/fd1a386c1e06fcbc9c06017d18e4fe5f.svg
Geon Choi
checkone
user
checkone
691e88213c64d32b0364590d
Hangyul Yoon
claimed_verified
2025-11-20T08:48:59.480Z
62a4d58e81a4b10e93064ad6
/avatars/744d5cbc1745a26b816a458260aba050.svg
hangyulyoon
hangyulmd
user
hangyulmd
691e88213c64d32b0364590e
Hyunju Shin
691e88213c64d32b0364590f
Hyunki Park
691e88213c64d32b03645910
Sang Hoon Seo
691e88213c64d32b03645911
Eunho Yang
691e88213c64d32b03645912
Edward Choi
Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset
The applicability of current lesion segmentation models for chest X-rays (CXRs) has been limited both by a small number of target labels and the reliance on long, detailed expert-level text inputs, creating a barrier to practical use. To address these limitations, we introduce a new paradigm: instruction-guided lesion segmentation (ILS), which is designed to segment diverse lesion types based on simple, user-friendly instructions. Under this paradigm, we construct MIMIC-ILS, the first large-scale instruction-answer dataset for CXR lesion segmentation, using our fully automated multimodal pipeline that generates annotations from chest X-ray images and their corresponding reports. MIMIC-ILS contains 1.1M instruction-answer pairs derived from 192K images and 91K unique segmentation masks, covering seven major lesion types. To empirically demonstrate its utility, we introduce ROSALIA, a vision-language model fine-tuned on MIMIC-ILS. ROSALIA can segment diverse lesions and provide textual explanations in response to user instructions. The model achieves high segmentation and textual accuracy in our newly proposed task, highlighting the effectiveness of our pipeline and the value of MIMIC-ILS as a foundational resource for pixel-level CXR lesion grounding.
A new instruction-guided lesion segmentation paradigm using a large-scale dataset and a vision-language model enables diverse CXR lesion segmentation with simple instructions.
instruction-guided lesion segmentation MIMIC-ILS multimodal pipeline vision-language model pixel-level CXR lesion grounding