Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models

About

Recently, unified multimodal models (UMMs) have made remarkable progress in integrating visual understanding and generation, demonstrating strong potential for complex text-to-image (T2I) tasks. Despite their theoretical promise, a persistent capability gap exists: UMMs typically exhibit superior visual understanding but comparatively weaker generative capabilities. This discrepancy arises largely from the intrinsic decoupling between the understanding and generation processes. While a UMM can accurately interpret fine-grained visual details, it often struggles to produce semantically coherent images from complex textual prompts. To address this challenge, we explore UMMs' internal understanding capability to enhance generation quality. We propose a token-level intrinsic text-image alignment reward mechanism, GvU, enabling the UMM to act simultaneously as teacher and student: it evaluates its own outputs using the understanding branch to guide the generations accordingly. Building upon this, we design a self-supervised reinforcement learning framework, allowing UMMs to iteratively improve their generation quality through understanding-based intrinsic reward signals--without reliance on external supervision. Experimental results show that our method substantially boosts UMMs' generation, which in turn strengthens their fine-grained visual understanding, narrowing the capability gap between UMMs' visual understanding and generation.

Jiadong Pan, Liang Li, Yuxin Peng, Yu-Ming Tang, Shuohuan Wang, Yu Sun, Hua Wu, Qingming Huang, Haifeng Wang• 2026

Related benchmarks

TaskDatasetResultRank
Object Hallucination EvaluationPOPE
Accuracy86.3
1455
Multimodal UnderstandingMMBench--
637
Visual Question AnsweringGQA
Accuracy61.7
505
Multimodal UnderstandingSEED-Bench
Accuracy73.9
343
Text-to-Image GenerationDPG-Bench
Overall Score85.68
265
Document Visual Question AnsweringDocVQA
ANLS88.4
263
Optical Character RecognitionOCRBench
Score709
232
Text-to-Image GenerationGenEval 1.0 (test)
Overall Score84
85
Text-to-Image GenerationGenEval++
Color Accuracy30
45
Showing 9 of 9 rows

Other info

Follow for update