Mirage2Matter: A Physically Grounded Gaussian World Model from Video
About
The scalability of embodied intelligence is fundamentally constrained by the scarcity of real-world interaction data. While simulation platforms provide a promising alternative, existing approaches often suffer from a substantial visual and physical gap to real environments and rely on expensive sensors, precise robot calibration, or depth measurements, limiting their practicality at scale. We present Simulate Anything, a graphics-driven world modeling and simulation framework that enables efficient generation of high-fidelity embodied training data using only multi-view environment videos and off-the-shelf assets. Our approach reconstructs real-world environments into a photorealistic scene representation using 3D Gaussian Splatting (3DGS), seamlessly capturing fine-grained geometry and appearance from video. We then leverage generative models to recover a physically realistic representation and integrate it into a simulation environment via a precision calibration target, enabling accurate scale alignment between the reconstructed scene and the real world. Together, these components provide a unified, editable, and physically grounded world model. Vision Language Action (VLA) models trained on our simulated data achieve strong zero-shot performance on downstream tasks, matching or even surpassing results obtained with real-world data, highlighting the potential of reconstruction-driven world modeling for scalable and practical embodied intelligence training.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Real-robot grasping | Real-robot grasping Banana 1.0 (test) | Success Rate0.8 | 4 | |
| Real-robot grasping | Real-robot grasping Croissant 1.0 (test) | Success Rate86.7 | 4 | |
| Press Button | Real-robot (test) | Success Rate93.3 | 2 | |
| Push/Pull Objects | Real-robot (test) | Success Rate73.3 | 2 |