OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs.
  Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics.
  Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists.
698c7b82eb12ea7453916913
Feilong Tang
698c7b82eb12ea7453916914
Xiang An
claimed_verified
2026-02-17T15:50:44.091Z
6478679d7b370854241b2ad8
https://cdn-avatars.huggingface.co/v1/production/uploads/6478679d7b370854241b2ad8/dBczWYYdfEt9tQcnVGhQk.jpeg
xiangan
xiangan
user
xiangan
698c7b82eb12ea7453916915
Yunyao Yan
698c7b82eb12ea7453916916
Yin Xie
698c7b82eb12ea7453916917
Bin Qin
698c7b82eb12ea7453916918
Kaicheng Yang
698c7b82eb12ea7453916919
Yifei Shen
698c7b82eb12ea745391691a
Yuanhan Zhang
698c7b82eb12ea745391691b
Chunyuan Li
698c7b82eb12ea745391691c
Shikun Feng
698c7b82eb12ea745391691d
Changrui Chen
698c7b82eb12ea745391691e
Huajie Tan
698c7b82eb12ea745391691f
Ming Hu
698c7b82eb12ea7453916920
Manyuan Zhang
698c7b82eb12ea7453916921
Bo Li
698c7b82eb12ea7453916922
Ziyong Feng
698c7b82eb12ea7453916923
Ziwei Liu
698c7b82eb12ea7453916924
Zongyuan Ge
698c7b82eb12ea7453916925
Jiankang Deng
OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs.
  Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics.
  Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists.
Visual understanding can be improved by aligning architectures with information-theoretic principles of video compression, using sparsity-driven encoding that outperforms traditional approaches in efficiency and accuracy.
artificial general intelligence compression problem resonance deep learning visual signals discriminative information pixel grids compute optimization video compression Codec Patchification 3D RoPE cluster discrimination objective semantic concepts object permanence motion dynamics LLM vision backbones visual tokens pretraining data video understanding sparsity-driven encoding