Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
69b3e5c30ee19b1e3a541e5e
Umberto Cappellazzo
claimed_verified
2026-03-13T16:50:02.901Z
64903f017b630c141867877f
https://cdn-avatars.huggingface.co/v1/production/uploads/64903f017b630c141867877f/IvVHLmY95_3J7Jd-E5Lfi.jpeg
Umberto Cappellazzo
hisoka94
user
hisoka94
69b3e5c30ee19b1e3a541e5f
Stavros Petridis
69b3e5c30ee19b1e3a541e60
Maja Pantic
Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognition
Audio-Visual Speech Recognition (AVSR) leverages both acoustic and visual information for robust recognition under noise. However, how models balance these modalities remains unclear. We present Dr. SHAP-AV, a framework using Shapley values to analyze modality contributions in AVSR. Through experiments on six models across two benchmarks and varying SNR levels, we introduce three analyses: Global SHAP for overall modality balance, Generative SHAP for contribution dynamics during decoding, and Temporal Alignment SHAP for input-output correspondence. Our findings reveal that models shift toward visual reliance under noise yet maintain high audio contributions even under severe degradation. Modality balance evolves during generation, temporal alignment holds under noise, and SNR is the dominant factor driving modality weighting. These findings expose a persistent audio bias, motivating ad-hoc modality-weighting mechanisms and Shapley-based attribution as a standard AVSR diagnostic.
Dr. SHAP-AV framework uses Shapley values to analyze modality contributions in audio-visual speech recognition, revealing how models balance acoustic and visual information under varying noise conditions.
Shapley values audio-visual speech recognition modality balance generative modeling temporal alignment signal-to-noise ratio attribution analysis