Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large
  Language Models
Large language models (LLMs) have recently achieved impressive results in
speech recognition across multiple modalities, including Auditory Speech
Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Speech
Recognition (AVSR). Despite this progress, current LLM-based approaches
typically address each task independently, training separate models that raise
computational and deployment resource use while missing potential cross-task
synergies. They also rely on fixed-rate token compression, which restricts
flexibility in balancing accuracy with efficiency. These limitations highlight
the need for a unified framework that can support ASR, VSR, and AVSR while
enabling elastic inference. To this end, we present Omni-AVSR, a unified
audio-visual LLM that combines efficient multi-granularity training with
parameter-efficient adaptation. Specifically, we adapt the matryoshka
representation learning paradigm to efficiently train across multiple audio and
visual granularities, reducing its inherent training resource use. Furthermore,
we explore three LoRA-based strategies for adapting the backbone LLM, balancing
shared and task-specific specialization. Experiments on LRS2 and LRS3 show that
Omni-AVSR achieves comparable or superior accuracy to state-of-the-art
baselines while training a single model at substantially lower training and
deployment resource use. The model also remains robust under acoustic noise,
and we analyze its scaling behavior as LLM size increases, providing insights
into the trade-off between performance and efficiency.
6912f39fa644ba07c499c861
Umberto Cappellazzo
claimed_verified
2025-11-11T19:43:13.094Z
64903f017b630c141867877f
https://cdn-avatars.huggingface.co/v1/production/uploads/64903f017b630c141867877f/IvVHLmY95_3J7Jd-E5Lfi.jpeg
Umberto Cappellazzo
hisoka94
user
hisoka94
6912f39fa644ba07c499c862
Xubo Liu
6912f39fa644ba07c499c863
Pingchuan Ma
6912f39fa644ba07c499c864
Stavros Petridis
6912f39fa644ba07c499c865
Maja Pantic
Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large
  Language Models
Large language models (LLMs) have recently achieved impressive results in
speech recognition across multiple modalities, including Auditory Speech
Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Speech
Recognition (AVSR). Despite this progress, current LLM-based approaches
typically address each task independently, training separate models that raise
computational and deployment resource use while missing potential cross-task
synergies. They also rely on fixed-rate token compression, which restricts
flexibility in balancing accuracy with efficiency. These limitations highlight
the need for a unified framework that can support ASR, VSR, and AVSR while
enabling elastic inference. To this end, we present Omni-AVSR, a unified
audio-visual LLM that combines efficient multi-granularity training with
parameter-efficient adaptation. Specifically, we adapt the matryoshka
representation learning paradigm to efficiently train across multiple audio and
visual granularities, reducing its inherent training resource use. Furthermore,
we explore three LoRA-based strategies for adapting the backbone LLM, balancing
shared and task-specific specialization. Experiments on LRS2 and LRS3 show that
Omni-AVSR achieves comparable or superior accuracy to state-of-the-art
baselines while training a single model at substantially lower training and
deployment resource use. The model also remains robust under acoustic noise,
and we analyze its scaling behavior as LLM size increases, providing insights
into the trade-off between performance and efficiency.
Omni-AVSR is a unified audio-visual LLM that efficiently supports ASR, VSR, and AVSR through multi-granularity training and parameter-efficient adaptation, achieving high accuracy with reduced resource use.
LLMs Auditory Speech Recognition (ASR) Visual Speech Recognition (VSR) Audio-Visual Speech Recognition (AVSR) matryoshka representation learning multi-granularity training parameter-efficient adaptation LoRA LRS2 LRS3 acoustic noise