Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs
As Large Language Models (LLMs) expand beyond text, integrating speech as a native modality has given rise to SpeechLLMs, which aim to translate spoken language directly, thereby bypassing traditional transcription-based pipelines. Whether this integration improves speech-to-text translation quality over established cascaded architectures, however, remains an open question. We present Hearing to Translate, the first comprehensive test suite rigorously benchmarking 5 state-of-the-art SpeechLLMs against 16 strong direct and cascade systems that couple leading speech foundation models (SFM), with multilingual LLMs. Our analysis spans 16 benchmarks, 13 language pairs, and 9 challenging conditions, including disfluent, noisy, and long-form speech. Across this extensive evaluation, we find that cascaded systems remain the most reliable overall, while current SpeechLLMs only match cascades in selected settings and SFMs lag behind both, highlighting that integrating an LLM, either within the model or in a pipeline, is essential for high-quality speech translation.
6945080bfbf17e708e1861c0
Sara Papi
claimed_verified
2025-12-19T08:57:12.964Z
66309b3833ccd9e68c5d5171
https://cdn-avatars.huggingface.co/v1/production/uploads/66309b3833ccd9e68c5d5171/UGt7pZazJHhtwjg0iiyBu.jpeg
Sara Papi
spapi
user
spapi
6945080bfbf17e708e1861c1
Javier Garcia Gilabert
claimed_verified
2025-12-19T10:49:52.784Z
647224b7609ae9f5636b3f32
https://cdn-avatars.huggingface.co/v1/production/uploads/647224b7609ae9f5636b3f32/-GJ6KFEvgTHdqJURl_I_J.jpeg
Javier García Gilabert
javi8979
user
javi8979
6945080bfbf17e708e1861c2
Zachary Hopton
claimed_verified
2025-12-19T16:25:48.414Z
66597b2ff149ff06ba57ed76
/avatars/a64a424bb4075b5cab33d5ba8eaacd44.svg
Zachary Hopton
zhopto3
user
zhopto3
6945080bfbf17e708e1861c3
Vilém Zouhar
claimed_verified
2026-01-09T08:37:06.687Z
6304ece07424d937fa35fb98
https://cdn-avatars.huggingface.co/v1/production/uploads/6304ece07424d937fa35fb98/Ct4fi1SpLUwhrGDYsAcVB.jpeg
Vilém Zouhar
zouharvi
user
zouharvi
6945080bfbf17e708e1861c4
Carlos Escolano
6945080bfbf17e708e1861c5
Gerard I. Gállego
6945080bfbf17e708e1861c6
Jorge Iranzo-Sánchez
6945080bfbf17e708e1861c7
Ahrii Kim
6945080bfbf17e708e1861c8
Dominik Macháček
6945080bfbf17e708e1861c9
Patricia Schmidtova
6945080bfbf17e708e1861ca
Maike Züfle
Hearing to Translate: The Effectiveness of Speech Modality Integration into LLMs
As Large Language Models (LLMs) expand beyond text, integrating speech as a native modality has given rise to SpeechLLMs, which aim to translate spoken language directly, thereby bypassing traditional transcription-based pipelines. Whether this integration improves speech-to-text translation quality over established cascaded architectures, however, remains an open question. We present Hearing to Translate, the first comprehensive test suite rigorously benchmarking 5 state-of-the-art SpeechLLMs against 16 strong direct and cascade systems that couple leading speech foundation models (SFM), with multilingual LLMs. Our analysis spans 16 benchmarks, 13 language pairs, and 9 challenging conditions, including disfluent, noisy, and long-form speech. Across this extensive evaluation, we find that cascaded systems remain the most reliable overall, while current SpeechLLMs only match cascades in selected settings and SFMs lag behind both, highlighting that integrating an LLM, either within the model or in a pipeline, is essential for high-quality speech translation.
Hearing to Translate benchmarks SpeechLLMs and cascaded systems for speech-to-text translation, finding that cascaded systems are more reliable overall and highlighting the importance of integrating LLMs for high-quality translation.
Large Language Models SpeechLLMs speech-to-text translation transcription-based pipelines cascaded architectures speech foundation models multilingual LLMs disfluent speech noisy speech long-form speech