Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization
Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs. Code and checkpoints will be released publicly at https://github.com/lucadellalib/dycast.
69856ae14ad556f294b7ec20
Luca Della Libera
claimed_verified
2026-02-10T09:07:53.312Z
63195d0582e7eec0eac040e3
https://cdn-avatars.huggingface.co/v1/production/uploads/63195d0582e7eec0eac040e3/0tXOYkMfmv9e53zBWgqz7.png
Luca Della Libera
lucadellalib
user
lucadellalib
69856ae14ad556f294b7ec21
Cem Subakan
69856ae14ad556f294b7ec22
Mirco Ravanelli
Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization
Neural audio codecs are at the core of modern conversational speech technologies, converting continuous speech into sequences of discrete tokens that can be processed by LLMs. However, existing codecs typically operate at fixed frame rates, allocating tokens uniformly in time and producing unnecessarily long sequences. In this work, we introduce DyCAST, a Dynamic Character-Aligned Speech Tokenizer that enables variable-frame-rate tokenization through soft character-level alignment and explicit duration modeling. DyCAST learns to associate tokens with character-level linguistic units during training and supports alignment-free inference with direct control over token durations at decoding time. To improve speech resynthesis quality at low frame rates, we further introduce a retrieval-augmented decoding mechanism that enhances reconstruction fidelity without increasing bitrate. Experiments show that DyCAST achieves competitive speech resynthesis quality and downstream performance while using significantly fewer tokens than fixed-frame-rate codecs. Code and checkpoints will be released publicly at https://github.com/lucadellalib/dycast.
DyCAST is a dynamic speech tokenizer that uses soft character-level alignment and duration modeling to enable variable-frame-rate tokenization, improving speech resynthesis quality with fewer tokens than traditional fixed-frame-rate codecs.
neural audio codecs conversational speech technologies discrete tokens LLMs fixed frame rates variable-frame-rate tokenization soft character-level alignment explicit duration modeling character-level linguistic units alignment-free inference retrieval-augmented decoding speech resynthesis downstream performance