Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

About

We present JanusFlow, a powerful framework that unifies image understanding and generation in a single model. JanusFlow introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling. Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications. To further improve the performance of our unified model, we adopt two key strategies: (i) decoupling the understanding and generation encoders, and (ii) aligning their representations during unified training. Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks. This work represents a step toward more efficient and versatile vision-language models.

Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, Chong Ruan• 2024

Related benchmarks

TaskDatasetResultRank
Object Hallucination EvaluationPOPE
Accuracy88
2056
Visual Question AnsweringTextVQA
Accuracy55.5
1455
Visual Question AnsweringGQA
Accuracy60.3
1445
Text-to-Image GenerationGenEval
Overall Score63
914
Multimodal UnderstandingMMBench
Accuracy74.9
887
Multimodal UnderstandingMM-Vet
MM-Vet Score31.8
664
Visual Question AnsweringChartQA
Accuracy64.6
620
Text-to-Image GenerationGenEval
Overall Score63
581
Multimodal UnderstandingSEED-Bench
Accuracy70.5
571
Multimodal UnderstandingMMStar
Accuracy40.6
511
Showing 10 of 82 rows
...

Other info

Code

Follow for update