Next-Embedding Prediction Makes Strong Vision Learners
Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.
6944c39ffbf17e708e18605d
Sihan Xu
claimed_verified
2025-12-19T08:57:39.350Z
63f233820a16587ea967adc2
https://cdn-avatars.huggingface.co/v1/production/uploads/63f233820a16587ea967adc2/1nSoZofPV7UseXzjI2qAH.png
Sihan XU
sihanxu
user
sihanxu
6944c39ffbf17e708e18605e
Ziqiao Ma
claimed_verified
2025-12-22T10:59:41.501Z
630cfc45b66f088d547b2768
https://cdn-avatars.huggingface.co/v1/production/uploads/630cfc45b66f088d547b2768/9-dMts2xFVbPmPHJGGqBx.png
Martin Ziqiao Ma
marstin
user
marstin
6944c39ffbf17e708e18605f
Wenhao Chai
6944c39ffbf17e708e186060
Xuweiyi Chen
6944c39ffbf17e708e186061
Weiyang Jin
claimed_verified
2025-12-19T09:56:57.800Z
66608add236f958513d21d2e
/avatars/53eca0891c98cbb93be899885160a983.svg
Weiyang Jin
Wayne-King
user
Wayne-King
6944c39ffbf17e708e186062
Joyce Chai
6944c39ffbf17e708e186063
Saining Xie
6944c39ffbf17e708e186064
Stella X. Yu
Next-Embedding Prediction Makes Strong Vision Learners
Inspired by the success of generative pretraining in natural language, we ask whether the same principles can yield strong self-supervised visual learners. Instead of training models to output features for downstream use, we train them to generate embeddings to perform predictive tasks directly. This work explores such a shift from learning representations to learning models. Specifically, models learn to predict future patch embeddings conditioned on past ones, using causal masking and stop gradient, which we refer to as Next-Embedding Predictive Autoregression (NEPA). We demonstrate that a simple Transformer pretrained on ImageNet-1k with next embedding prediction as its sole learning objective is effective - no pixel reconstruction, discrete tokens, contrastive loss, or task-specific heads. This formulation retains architectural simplicity and scalability, without requiring additional design complexity. NEPA achieves strong results across tasks, attaining 83.8% and 85.3% top-1 accuracy on ImageNet-1K with ViT-B and ViT-L backbones after fine-tuning, and transferring effectively to semantic segmentation on ADE20K. We believe generative pretraining from embeddings provides a simple, scalable, and potentially modality-agnostic alternative to visual self-supervised learning.
Generative pretraining using next embedding prediction outperforms traditional self-supervised methods in visual learning tasks, achieving high accuracy on ImageNet and effective transfer to semantic segmentation.
generative pretraining predictive tasks Next-Embedding Predictive Autoregression (NEPA) causal masking stop gradient Transformer ImageNet-1k top-1 accuracy ViT-B ViT-L semantic segmentation ADE20K