MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models
Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. To this end, we first propose CAT (Causal Audio Tokenizer with Transformer), a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction. Building on the CAT architecture, we develop MOSS-Audio-Tokenizer, a large-scale audio tokenizer featuring 1.6 billion parameters, pre-trained on 3 million hours of diverse, general audio data. We show that this simple, fully end-to-end approach built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, MOSS-Audio-Tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging the discrete tokens from our model, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, MOSS-Audio-Tokenizer enables competitive ASR performance without auxiliary encoders. Our findings position the CAT architecture as a unified, scalable interface for the next generation of native audio foundation models.
698d585765c0d15a6d1621fe
Yitian Gong
claimed_verified
2026-02-12T13:26:16.742Z
66c893b2e51ba3009235b1c0
/avatars/5341d40b4b4caca2c145a46eb1754582.svg
yitian gong
fdugyt
user
fdugyt
698d585765c0d15a6d1621ff
Kuangwei Chen
698d585765c0d15a6d162200
Zhaoye Fei
claimed_verified
2026-02-13T09:37:26.561Z
629ef8544313a7c1dd671130
https://cdn-avatars.huggingface.co/v1/production/uploads/629ef8544313a7c1dd671130/i5xfHIgELcuO1Ew19ebTw.png
Zhaoye Fei
ngc7293
user
ngc7293
698d585765c0d15a6d162201
Xiaogui Yang
698d585765c0d15a6d162202
Ke Chen
698d585765c0d15a6d162203
Yang Wang
698d585765c0d15a6d162204
Kexin Huang
698d585765c0d15a6d162205
Mingshu Chen
698d585765c0d15a6d162206
Ruixiao Li
698d585765c0d15a6d162207
Qingyuan Cheng
698d585765c0d15a6d162208
Shimin Li
698d585765c0d15a6d162209
Xipeng Qiu
MOSS-Audio-Tokenizer: Scaling Audio Tokenizers for Future Audio Foundation Models
Discrete audio tokenizers are fundamental to empowering large language models with native audio processing and generation capabilities. Despite recent progress, existing approaches often rely on pretrained encoders, semantic distillation, or heterogeneous CNN-based architectures. These designs introduce fixed inductive biases that limit reconstruction fidelity and hinder effective scaling. In this paper, we argue that discrete audio tokenization should be learned fully end-to-end using a homogeneous and scalable architecture. To this end, we first propose CAT (Causal Audio Tokenizer with Transformer), a purely Transformer-based architecture that jointly optimizes the encoder, quantizer, and decoder from scratch for high-fidelity reconstruction. Building on the CAT architecture, we develop MOSS-Audio-Tokenizer, a large-scale audio tokenizer featuring 1.6 billion parameters, pre-trained on 3 million hours of diverse, general audio data. We show that this simple, fully end-to-end approach built from homogeneous, causal Transformer blocks scales gracefully and supports high-fidelity reconstruction across diverse audio domains. Across speech, sound, and music, MOSS-Audio-Tokenizer consistently outperforms prior codecs over a wide range of bitrates, while exhibiting predictable improvements with increased scale. Notably, leveraging the discrete tokens from our model, we develop the first purely autoregressive TTS model that surpasses prior non-autoregressive and cascaded systems. Furthermore, MOSS-Audio-Tokenizer enables competitive ASR performance without auxiliary encoders. Our findings position the CAT architecture as a unified, scalable interface for the next generation of native audio foundation models.
A fully end-to-end Transformer-based audio tokenizer architecture achieves high-fidelity reconstruction across diverse audio domains and enables superior text-to-speech and automatic speech recognition performance.
audio tokenization causal audio tokenizer Transformer architecture end-to-end learning quantizer encoder decoder discrete audio tokens autoregressive TTS automatic speech recognition codec pre-trained encoders semantic distillation heterogeneous CNN-based architectures