Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

About

Vision-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent works have begun to explore the incorporation of latent actions, abstract representations of motion between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Vision-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. We demonstrate that villa-X can generate latent action plans in a zero-shot fashion, even for unseen embodiments and open-vocabulary symbolic understanding. This capability enables villa-X to achieve superior performance across diverse simulation tasks in SIMPLER and on two real-world robotic setups involving both gripper and dexterous hand manipulation. These results establish villa-X as a principled and scalable paradigm for learning generalizable robot manipulation policies. We believe it provides a strong foundation for future research.

Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, Jianyu Chen, Jiang Bian• 2025

Related benchmarks

TaskDatasetResultRank
Robot ManipulationLIBERO
Goal Achievement91.5
700
Robotic ManipulationLIBERO
Spatial Success Rate97.5
314
Robot ManipulationSimplerEnv WidowX Robot tasks (test)
Success Rate (Spoon)48.3
79
Robot ManipulationSimplerEnv Google Robot tasks Visual Matching
Pick Coke Can Success Rate81.7
62
Robot ManipulationSimplerEnv WidowX
Success Rate: Put Spoon on Towel77.9
58
Robotic ManipulationSIMPLER Visual Matching WidowX robot
Put Spoon on Towel Score48.3
51
Robotic ManipulationLIBERO v1 (test)
Average Success Rate90.1
46
Robotic ManipulationLIBERO Spatial Object Goal Long
Overall Success Rate (Long)74.5
31
Robotic ManipulationSIMPLER Google Robot Visual Matching
PickCan Success Rate81.7
24
Robot ManipulationSimplerEnv Google Robot (test)
Pick Coke Can98.7
11
Showing 10 of 15 rows

Other info

Follow for update