Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding

About

Graphical User Interface (GUI) grounding maps natural language instructions to the visual coordinates of target elements and serves as a core capability for autonomous GUI agents. Recent reinforcement learning methods (e.g., GRPO) have achieved strong performance, but they rely on expensive multiple rollouts and suffer from sparse signals on hard samples. These limitations make on-policy self-distillation (OPSD), which provides dense token-level supervision from a single rollout, a promising alternative. However, its applicability to GUI grounding remains unexplored. In this paper, we present GUI-SD, the first OPSD framework tailored for GUI grounding. First, it constructs a visually enriched privileged context for the teacher using a target bounding box and a Gaussian soft mask, providing informative guidance without leaking exact coordinates. Second, it employs entropy-guided distillation, which adaptively weights tokens based on digit significance and teacher confidence, concentrating optimization on the most impactful and reliable positions. Extensive experiments on six representative GUI grounding benchmarks show that GUI-SD consistently outperforms GRPO-based methods and naive OPSD in both accuracy and training efficiency. Code and training data are available at https://zhangyan-ucas.github.io/GUI-SD/.

Yan Zhang, Daiqing Wu, Huawen Shen, Can Ma, Yu Zhou• 2026

Related benchmarks

TaskDatasetResultRank
GUI GroundingScreenSpot v2
Avg Accuracy95.1
371
GUI GroundingScreenSpot Pro
Accuracy60.7
195
GUI GroundingOSWorld-G--
144
GUI GroundingMMBench-GUI-L2
Accuracy86.7
43
GUI GroundingUI-Vision
Accuracy33.3
11
GUI GroundingOSWorld-G-Refine
Accuracy70.9
7
Showing 6 of 6 rows

Other info

Follow for update