Reconstruction Alignment Improves Unified Multimodal Models
Unified multimodal models (UMMs) unify visual understanding and generation
within a single architecture. However, conventional training relies on
image-text pairs (or sequences) whose captions are typically sparse and miss
fine-grained visual details--even when they use hundreds of words to describe a
simple image. We introduce Reconstruction Alignment (RecA), a
resource-efficient post-training method that leverages visual understanding
encoder embeddings as dense "text prompts," providing rich supervision without
captions. Concretely, RecA conditions a UMM on its own visual understanding
embeddings and optimizes it to reconstruct the input image with a
self-supervised reconstruction loss, thereby realigning understanding and
generation. Despite its simplicity, RecA is broadly applicable: across
autoregressive, masked-autoregressive, and diffusion-based UMMs, it
consistently improves generation and editing fidelity. With only 27 GPU-hours,
post-training with RecA substantially improves image generation performance on
GenEval (0.73rightarrow0.90) and DPGBench (80.93rightarrow88.15), while
also boosting editing benchmarks (ImgEdit 3.38rightarrow3.75, GEdit
6.94rightarrow7.25). Notably, RecA surpasses much larger open-source models
and applies broadly across diverse UMM architectures, establishing it as an
efficient and general post-training alignment strategy for UMMs
68c0e2033912ed54cf543276
Ji Xie
claimed_verified
2025-09-12T16:17:47.815Z
64e99fc07e2ec711a7138262
https://cdn-avatars.huggingface.co/v1/production/uploads/64e99fc07e2ec711a7138262/9-Y-ZFaUhDpi_8X4oRO5E.jpeg
Ji Xie
sanaka87
user
sanaka87
68c0e2033912ed54cf543277
Trevor Darrell
68c0e2033912ed54cf543278
Luke Zettlemoyer
68c0e2033912ed54cf543279
XuDong Wang
Reconstruction Alignment Improves Unified Multimodal Models
Unified multimodal models (UMMs) unify visual understanding and generation
within a single architecture. However, conventional training relies on
image-text pairs (or sequences) whose captions are typically sparse and miss
fine-grained visual details--even when they use hundreds of words to describe a
simple image. We introduce Reconstruction Alignment (RecA), a
resource-efficient post-training method that leverages visual understanding
encoder embeddings as dense "text prompts," providing rich supervision without
captions. Concretely, RecA conditions a UMM on its own visual understanding
embeddings and optimizes it to reconstruct the input image with a
self-supervised reconstruction loss, thereby realigning understanding and
generation. Despite its simplicity, RecA is broadly applicable: across
autoregressive, masked-autoregressive, and diffusion-based UMMs, it
consistently improves generation and editing fidelity. With only 27 GPU-hours,
post-training with RecA substantially improves image generation performance on
GenEval (0.73rightarrow0.90) and DPGBench (80.93rightarrow88.15), while
also boosting editing benchmarks (ImgEdit 3.38rightarrow3.75, GEdit
6.94rightarrow7.25). Notably, RecA surpasses much larger open-source models
and applies broadly across diverse UMM architectures, establishing it as an
efficient and general post-training alignment strategy for UMMs
Reconstruction Alignment (RecA) is a post-training method that enhances multimodal models by using visual embeddings as dense prompts, improving image generation and editing fidelity.
Unified multimodal models Reconstruction Alignment visual understanding encoder embeddings self-supervised reconstruction loss autoregressive masked-autoregressive diffusion-based GenEval DPGBench ImgEdit GEdit