Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization

About

The emergence of Multimodal Large Language Models (MLLMs) has propelled the development of autonomous agents that operate on Graphical User Interfaces (GUIs) using pure visual input. A fundamental challenge is robustly grounding natural language instructions. This requires a precise spatial alignment, which accurately locates the coordinates of each element, and, more critically, a correct semantic alignment, which matches the instructions to the functionally appropriate UI element. Although Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be effective at improving spatial alignment for these MLLMs, we find that inefficient exploration bottlenecks semantic alignment, which prevent models from learning difficult semantic associations. To address this exploration problem, we present Adaptive Exploration Policy Optimization (AEPO), a new policy optimization framework. AEPO employs a multi-answer generation strategy to enforce broader exploration, which is then guided by a theoretically grounded Adaptive Exploration Reward (AER) function derived from first principles of efficiency eta=U/C. Our AEPO-trained models, InfiGUI-G1-3B and InfiGUI-G1-7B, establish new state-of-the-art results across multiple challenging GUI grounding benchmarks, achieving significant relative improvements of up to 9.0% against the naive RLVR baseline on benchmarks designed to test generalization and semantic understanding. Resources are available at https://github.com/InfiXAI/InfiGUI-G1.

Yuhang Liu, Zeyu Liu, Shuanghe Zhu, Pengxiang Li, Congkai Xie, Jiasheng Wang, Xavier Hu, Xiaotian Han, Jianbo Yuan, Xinyao Wang, Shengyu Zhang, Hongxia Yang, Fei Wu• 2025

Related benchmarks

TaskDatasetResultRank
GUI GroundingScreenSpot Pro
Average Score5.19e+3
307
GUI GroundingScreenSpot v2
Avg Accuracy93.5
283
GUI GroundingMMBench-GUI L2 (test)
Average Error73.4
67
GUI GroundingUI-Vision
Average Score26.1
59
GUI GroundingScreenSpot Web V2
Text Accuracy97.9
55
GUI GroundingScreenSpot Mobile V2
Text Accuracy99
55
GUI GroundingScreenSpot Desktop V2
Text Accuracy94.3
55
GUI GroundingUI-Vision (test)
Basic Score36.2
43
UI GroundingUI-Vision--
24
GUI Agent NavigationMMBench-GUI
Windows (Basic) Score82.7
16
Showing 10 of 10 rows

Other info

Follow for update