Taming Modality Entanglement in Continual Audio-Visual Segmentation
Recently, significant progress has been made in multi-modal continual
learning, aiming to learn new tasks sequentially in multi-modal settings while
preserving performance on previously learned ones. However, existing methods
mainly focus on coarse-grained tasks, with limitations in addressing modality
entanglement in fine-grained continual learning settings. To bridge this gap,
we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to
continuously segment new classes guided by audio. Through comprehensive
analysis, two critical challenges are identified: 1) multi-modal semantic
drift, where a sounding objects is labeled as background in sequential tasks;
2) co-occurrence confusion, where frequent co-occurring classes tend to be
confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework
is designed to address these challenges. Specifically, for multi-modal semantic
drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select
samples with high modal consistency for rehearsal. Meanwhile, for co-occurence
confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed,
allowing for the increase of rehearsal sample frequency of those confusable
classes during training process. Moreover, we construct three audio-visual
incremental scenarios to verify effectiveness of our method. Comprehensive
experiments demonstrate that our method significantly outperforms single-modal
continual learning methods.
68f9ab58b9b2e4ae046738b1
Yuyang Hong
claimed_verified
2025-10-25T14:34:02.018Z
66e03d7123e5f162e7e5c5ac
https://cdn-avatars.huggingface.co/v1/production/uploads/66e03d7123e5f162e7e5c5ac/PHHGlQKnZhr_BNpciB9Zz.jpeg
hongyuyang
hongyuyang23casia
user
hongyuyang23casia
68f9ab58b9b2e4ae046738b2
Qi Yang
claimed_verified
2025-11-28T13:49:14.176Z
643e45c4c639f8bc9727810a
https://cdn-avatars.huggingface.co/v1/production/uploads/643e45c4c639f8bc9727810a/BJR1cvSCxcqxr08iS7GsI.jpeg
YannQi
YannQi
user
YannQi
68f9ab58b9b2e4ae046738b3
Tao Zhang
claimed_verified
2025-12-12T09:18:38.602Z
66ceeab7528c51fee3d2a25e
/avatars/23833573f668abc1bf343cca79b3d7a0.svg
zhangtao
casiatao
user
casiatao
68f9ab58b9b2e4ae046738b4
Zili Wang
68f9ab58b9b2e4ae046738b5
Zhaojin Fu
68f9ab58b9b2e4ae046738b6
Kun Ding
68f9ab58b9b2e4ae046738b7
Bin Fan
68f9ab58b9b2e4ae046738b8
Shiming Xiang
Taming Modality Entanglement in Continual Audio-Visual Segmentation
Recently, significant progress has been made in multi-modal continual
learning, aiming to learn new tasks sequentially in multi-modal settings while
preserving performance on previously learned ones. However, existing methods
mainly focus on coarse-grained tasks, with limitations in addressing modality
entanglement in fine-grained continual learning settings. To bridge this gap,
we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to
continuously segment new classes guided by audio. Through comprehensive
analysis, two critical challenges are identified: 1) multi-modal semantic
drift, where a sounding objects is labeled as background in sequential tasks;
2) co-occurrence confusion, where frequent co-occurring classes tend to be
confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework
is designed to address these challenges. Specifically, for multi-modal semantic
drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select
samples with high modal consistency for rehearsal. Meanwhile, for co-occurence
confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed,
allowing for the increase of rehearsal sample frequency of those confusable
classes during training process. Moreover, we construct three audio-visual
incremental scenarios to verify effectiveness of our method. Comprehensive
experiments demonstrate that our method significantly outperforms single-modal
continual learning methods.
A novel Continual Audio-Visual Segmentation (CAVS) task addresses challenges in multi-modal continual learning through a Collision-based Multi-modal Rehearsal (CMR) framework, improving performance over single-modal methods.
multi-modal continual learning Continual Audio-Visual Segmentation CAVS multi-modal semantic drift co-occurrence confusion Collision-based Multi-modal Rehearsal CMR Multi-modal Sample Selection MSS Collision-based Sample Rehearsal CSR