JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
We introduce a two-stage self-supervised framework that combines the Joint-Embedding Predictive Architecture (JEPA) with a Density Adaptive Attention Mechanism (DAAM) for learning robust speech representations. Stage~1 uses JEPA with DAAM to learn semantic audio features via masked prediction in latent space, fully decoupled from waveform reconstruction. Stage~2 leverages these representations for efficient tokenization using Finite Scalar Quantization (FSQ) and a mixed-radix packing scheme, followed by high-fidelity waveform reconstruction with a HiFi-GAN decoder. By integrating Gaussian mixture-based density-adaptive gating into the JEPA encoder, the model performs adaptive temporal feature selection and discovers hierarchical speech structure at a low frame rate of 2.5~Hz. The resulting tokens (47.5 tokens/sec) provide a reversible, highly compressed, and language-model-friendly representation that is competitive with, and often more efficient than, existing neural audio codecs.
6938ad16dfc35938ba129efa
Georgios Ioannides
6938ad16dfc35938ba129efb
Christos Constantinou
6938ad16dfc35938ba129efc
Aman Chadha
claimed_verified
2025-12-10T08:56:42.379Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
amanchadha
user
amanchadha
6938ad16dfc35938ba129efd
Aaron Elkins
6938ad16dfc35938ba129efe
Linsey Pang
6938ad16dfc35938ba129eff
Ravid Shwartz-Ziv
6938ad16dfc35938ba129f00
Yann LeCun
JEPA as a Neural Tokenizer: Learning Robust Speech Representations with Density Adaptive Attention
We introduce a two-stage self-supervised framework that combines the Joint-Embedding Predictive Architecture (JEPA) with a Density Adaptive Attention Mechanism (DAAM) for learning robust speech representations. Stage~1 uses JEPA with DAAM to learn semantic audio features via masked prediction in latent space, fully decoupled from waveform reconstruction. Stage~2 leverages these representations for efficient tokenization using Finite Scalar Quantization (FSQ) and a mixed-radix packing scheme, followed by high-fidelity waveform reconstruction with a HiFi-GAN decoder. By integrating Gaussian mixture-based density-adaptive gating into the JEPA encoder, the model performs adaptive temporal feature selection and discovers hierarchical speech structure at a low frame rate of 2.5~Hz. The resulting tokens (47.5 tokens/sec) provide a reversible, highly compressed, and language-model-friendly representation that is competitive with, and often more efficient than, existing neural audio codecs.
A two-stage self-supervised framework combines JEPA with DAAM to learn robust speech representations, using masked prediction, FSQ, mixed-radix packing, and HiFi-GAN for efficient tokenization and waveform reconstruction.
Joint-Embedding Predictive Architecture (JEPA) Density Adaptive Attention Mechanism (DAAM) masked prediction latent space Finite Scalar Quantization (FSQ) mixed-radix packing HiFi-GAN Gaussian mixture-based density-adaptive gating adaptive temporal feature selection hierarchical speech structure