Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

UI2Code^N: UI-to-Code Generation as Interactive Visual Optimization

About

UI-to-code aims to translate UI screenshots into executable front-end code. Despite progress with vision-language models (VLMs), most existing methods formulate UI-to-code as a single-pass generation, which mismatches real-world UI development that is inherently iterative and feedback-driven. We reformulate UI-to-code as an interactive visual optimization problem, where code generation is embedded in a closed-loop process of execution, visual inspection, and iterative refinement driven by rendered visual feedback. To address the non-differentiability of visual objectives and the noise of absolute visual evaluators, we propose Relative Visual Policy Optimization (RVPO), a preference-based reinforcement learning method that optimizes relative visual rankings among rendered candidates under execution feedback. We instantiate this paradigm in UI2Code^N, an open-source 9B model trained via continual pre-training, supervised fine-tuning, and reinforcement learning. Experiments demonstrate state-of-the-art performance on UI drafting, UI polishing, and UI editing benchmarks, even outperforming larger models, with performance consistently improving through iterative visual optimization. Our code and models are available at https://github.com/zai-org/UI2Code_N.

Zhen Yang, Wenyi Hong, Mingde Xu, Xinyue Fan, Weihan Wang, Jiale Cheng, Xiaotao Gu, Jie Tang• 2025

Related benchmarks

TaskDatasetResultRank
UI DraftingFlame
VLM Score95
23
UI DraftingUI2Code-Real
VLM Score76.5
23
UI PolishingUIPolish-Synthetic
VLM Score94
23
UI DraftingWeb2Code
VLM Score92.5
23
UI DraftingDesign2Code
VLM Score88.6
23
UI PolishingUIPolish Real
VLM Score80
23
UI EditingUIEdit-Bench
Edit Correctness4.94
6
UI-to-Code generationDesign2Code
Block Accuracy88.7
6
UI-to-Code generationWebCode2M Long
CLIP Similarity0.79
4
UI-to-Code generationDesign2Code HARD 80 samples
Win Rate96.3
4
Showing 10 of 12 rows

Other info

GitHub

Follow for update