Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
69cce2df3f84a0971be66ce1
Shashi Kumar
69cce2df3f84a0971be66ce2
Esaú Villatoro-Tello
69cce2df3f84a0971be66ce3
Sergio Burdisso
69cce2df3f84a0971be66ce4
Kadri Hacioglu
69cce2df3f84a0971be66ce5
Thibault Bañeras-Roux
69cce2df3f84a0971be66ce6
Hasindri Watawana
69cce2df3f84a0971be66ce7
Dairazalia Sanchez-Cortes
69cce2df3f84a0971be66ce8
Srikanth Madikeri
69cce2df3f84a0971be66ce9
Petr Motlicek
69cce2df3f84a0971be66cea
Andreas Stolcke
Distilling Conversations: Abstract Compression of Conversational Audio Context for LLM-based ASR
Standard LLM-based speech recognition systems typically process utterances in isolation, limiting their ability to leverage conversational context. In this work, we study whether multimodal context from prior turns improves LLM-based ASR and how to represent that context efficiently. We find that, after supervised multi-turn training, conversational context mainly helps with the recognition of contextual entities. However, conditioning on raw context is expensive because the prior-turn audio token sequence grows rapidly with conversation length. To address this, we propose Abstract Compression, which replaces the audio portion of prior turns with a fixed number of learned latent tokens while retaining corresponding transcripts explicitly. On both in-domain and out-of-domain test sets, the compressed model recovers part of the gains of raw-context conditioning with a smaller prior-turn audio footprint. We also provide targeted analyses of the compression setup and its trade-offs.
Multimodal conversational context improves LLM-based speech recognition, and abstract compression reduces computational overhead by replacing audio sequences with learned latent tokens while preserving transcripts.
LLM-based speech recognition multimodal context conversational context supervised multi-turn training audio token sequence latent tokens abstract compression