NLE: Non-autoregressive LLM-based ASR by Transcript Editing
While autoregressive (AR) LLM-based ASR systems achieve strong accuracy, their sequential decoding limits parallelism and incurs high latency. We propose NLE, a non-autoregressive (NAR) approach that formulates speech recognition as conditional transcript editing, enabling fully parallel prediction. NLE extracts acoustic embeddings and an initial hypothesis from a pretrained speech encoder, then refines the hypothesis using a bidirectional LLM editor trained with a latent alignment objective. An interleaved padding strategy exploits the identity mapping bias of Transformers, allowing the model to focus on corrections rather than full reconstruction. On the Open ASR leaderboard, NLE++ achieves 5.67% average WER with an RTFx (inverse real-time factor) of 1630. In single-utterance scenarios, NLE achieves 27x speedup over the AR baseline, making it suitable for real-time applications.
69aff4629dbab2a62ec28055
Avihu Dekel
69aff4629dbab2a62ec28056
Samuel Thomas
69aff4629dbab2a62ec28057
Takashi Fukada
69aff4629dbab2a62ec28058
George Saon
NLE: Non-autoregressive LLM-based ASR by Transcript Editing
While autoregressive (AR) LLM-based ASR systems achieve strong accuracy, their sequential decoding limits parallelism and incurs high latency. We propose NLE, a non-autoregressive (NAR) approach that formulates speech recognition as conditional transcript editing, enabling fully parallel prediction. NLE extracts acoustic embeddings and an initial hypothesis from a pretrained speech encoder, then refines the hypothesis using a bidirectional LLM editor trained with a latent alignment objective. An interleaved padding strategy exploits the identity mapping bias of Transformers, allowing the model to focus on corrections rather than full reconstruction. On the Open ASR leaderboard, NLE++ achieves 5.67% average WER with an RTFx (inverse real-time factor) of 1630. In single-utterance scenarios, NLE achieves 27x speedup over the AR baseline, making it suitable for real-time applications.
A non-autoregressive speech recognition approach formulates acoustic-to-text conversion as conditional transcript editing using a bidirectional language model editor with latent alignment training and interleaved padding for improved efficiency.
autoregressive non-autoregressive speech recognition conditional transcript editing acoustic embeddings pretrained speech encoder bidirectional LLM editor latent alignment objective interleaved padding Transformer real-time factor WER