VisCritic: Visual State Comparison as Process Reward for GUI Agents
About
GUI agents powered by vision-language models show strong potential for automating digital tasks, yet frequently fail in long-horizon scenarios due to the absence of step-level verification. Existing process reward models verify actions through textual reasoning alone, missing the visual nature of GUI state changes. We introduce VisCritic, a visual process reward framework that verifies agent actions by directly comparing pre-action and post-action screenshots in visual feature space. VisCritic employs a Siamese vision transformer to extract change-aware representations, coupled with an Action-Aware Critic Head that jointly evaluates action success, task progress, and error type. A critic-training data construction pipeline generates weakly supervised samples from existing trajectories without additional human labels for critic training. Experiments and offline analyses across five benchmarks demonstrate that VisCritic serves as a plug-and-play enhancement for diverse GUI agents, generally improving benchmark metrics while providing visual diagnostic cues.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Web Agent | WebArena | Success Rate29.4 | 56 | |
| Mobile UI Control | AndroidWorld | Overall Task Success Rate29.8 | 45 | |
| Web navigation | Mind2Web | -- | 41 | |
| GUI Agent Interaction | OSWorld | -- | 16 | |
| Offline Action Consistency Diagnostic | AITW | Action Accuracy74.2 | 15 | |
| Action Success Prediction | AndroidWorld (smallest held-out split) | Precision87.1 | 3 |