DREAM: Where Visual Understanding Meets Text-to-Image Generation
Unifying visual representation learning and text-to-image (T2I) generation within a single model remains a central challenge in multimodal learning. We introduce DREAM, a unified framework that jointly optimizes discriminative and generative objectives, while learning strong visual representations. DREAM is built on two key techniques: During training, Masking Warmup, a progressive masking schedule, begins with minimal masking to establish the contrastive alignment necessary for representation learning, then gradually transitions to full masking for stable generative training. At inference, DREAM employs Semantically Aligned Decoding to align partially masked image candidates with the target text and select the best one for further decoding, improving text-image fidelity (+6.3%) without external rerankers. Trained solely on CC12M, DREAM achieves 72.7% ImageNet linear-probing accuracy (+1.1% over CLIP) and an FID of 4.25 (+6.2% over FLUID), with consistent gains in few-shot classification, semantic segmentation, and depth estimation. These results demonstrate that discriminative and generative objectives can be synergistic, allowing unified multimodal models that excel at both visual understanding and generation.
69a7a0422833ff1122ff642d
Chao Li
69a7a0422833ff1122ff642e
Tianhong Li
69a7a0422833ff1122ff642f
Sai Vidyaranya Nuthalapati
69a7a0422833ff1122ff6430
Hong-You Chen
69a7a0422833ff1122ff6431
Satya Narayan Shukla
69a7a0422833ff1122ff6432
Yonghuan Yang
69a7a0422833ff1122ff6433
Jun Xiao
69a7a0422833ff1122ff6434
Xiangjun Fan
69a7a0422833ff1122ff6435
Aashu Singh
69a7a0422833ff1122ff6436
Dina Katabi
69a7a0422833ff1122ff6437
Shlok Kumar Mishra
claimed_verified
2026-03-05T09:59:25.381Z
6440e030608bf7e3ad72668b
/avatars/a8ecab384a928bccf2e13422a966bdd0.svg
Shlok Kumar Mishra
shlokm
user
shlokm
DREAM: Where Visual Understanding Meets Text-to-Image Generation
Unifying visual representation learning and text-to-image (T2I) generation within a single model remains a central challenge in multimodal learning. We introduce DREAM, a unified framework that jointly optimizes discriminative and generative objectives, while learning strong visual representations. DREAM is built on two key techniques: During training, Masking Warmup, a progressive masking schedule, begins with minimal masking to establish the contrastive alignment necessary for representation learning, then gradually transitions to full masking for stable generative training. At inference, DREAM employs Semantically Aligned Decoding to align partially masked image candidates with the target text and select the best one for further decoding, improving text-image fidelity (+6.3%) without external rerankers. Trained solely on CC12M, DREAM achieves 72.7% ImageNet linear-probing accuracy (+1.1% over CLIP) and an FID of 4.25 (+6.2% over FLUID), with consistent gains in few-shot classification, semantic segmentation, and depth estimation. These results demonstrate that discriminative and generative objectives can be synergistic, allowing unified multimodal models that excel at both visual understanding and generation.
DREAM is a unified multimodal framework that combines visual representation learning and text-to-image generation through progressive masking and semantically aligned decoding, achieving superior performance in both discriminative and generative tasks.
unified framework discriminative objectives generative objectives visual representations masking warmup progressive masking schedule contrastive alignment semantically aligned decoding text-to-image generation image classification few-shot classification semantic segmentation depth estimation