Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Reconstruction Alignment Improves Unified Multimodal Models

About

Unified multimodal models (UMMs) unify visual understanding and generation within a single architecture. However, conventional training relies on image-text pairs (or sequences) whose captions are typically sparse and miss fine-grained visual details--even when they use hundreds of words to describe a simple image. We introduce Reconstruction Alignment (RecA), a resource-efficient post-training method that leverages visual understanding encoder embeddings as dense "text prompts," providing rich supervision without captions. Concretely, RecA conditions a UMM on its own visual understanding embeddings and optimizes it to reconstruct the input image with a self-supervised reconstruction loss, thereby realigning understanding and generation. Despite its simplicity, RecA is broadly applicable: across autoregressive, masked-autoregressive, and diffusion-based UMMs, it consistently improves generation and editing fidelity. With only 27 GPU-hours, post-training with RecA substantially improves image generation performance on GenEval (0.73$\rightarrow$0.90) and DPGBench (80.93$\rightarrow$88.15), while also boosting editing benchmarks (ImgEdit 3.38$\rightarrow$3.75, GEdit 6.94$\rightarrow$7.25). Notably, RecA surpasses much larger open-source models and applies broadly across diverse UMM architectures, establishing it as an efficient and general post-training alignment strategy for UMMs

Ji Xie, Trevor Darrell, Luke Zettlemoyer, XuDong Wang• 2025

Related benchmarks

TaskDatasetResultRank
Object Hallucination EvaluationPOPE--
1455
Visual Question AnsweringGQA
Accuracy58.5
1249
Text-to-Image GenerationGenEval
Overall Score85.2
506
Text-to-Image GenerationGenEval
Overall Score86
391
Vision UnderstandingMMMU
Accuracy34.9
65
Multimodal UnderstandingMMMU
MMMU Score35.7
59
Visual UnderstandingMME
MME Score1.19e+3
54
Text-to-Image GenerationWISE
WISE Score0.4
48
Text-to-Image GenerationCompBench
Overall Score87.2
33
Text-to-Image GenerationDPGBench
DPGBench Overall Score86.5
24
Showing 10 of 14 rows

Other info

Follow for update