Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Towards Unified World Models for Visual Navigation via Memory-Augmented Planning and Foresight

About

Enabling embodied agents to imagine future states is essential for robust and generalizable visual navigation. Yet, state-of-the-art systems typically rely on modular designs that decouple navigation planning from visual world modeling, which often induces state-action misalignment and weak adaptability in novel or dynamic scenarios. We propose UniWM, a unified, memory-augmented world model that integrates egocentric visual foresight and planning within a single multimodal autoregressive backbone. UniWM explicitly grounds action selection in visually imagined outcomes, tightly aligning prediction with control. Meanwhile, a hierarchical memory mechanism fuses short-term perceptual cues with longer-term trajectory context, supporting stable and coherent reasoning over extended horizons. Extensive experiments on four challenging benchmarks (Go Stanford, ReCon, SCAND, HuRoN) and the 1X Humanoid Dataset show that UniWM improves navigation success rates by up to 30%, substantially reduces trajectory errors against strong baselines, generalizes zero-shot to the unseen TartanDrive dataset, and scales naturally to high-dimensional humanoid control. These results position UniWM as a principled step toward unified, imagination-driven embodied navigation. The code and models are available at https://github.com/F1y1113/UniWM.

Yifei Dong, Fengyi Wu, Guangyu Chen, Lingdong Kong, Xu Zhu, Qiyu Hu, Yuxuan Zhou, Jingdong Sun, Jun-Yan He, Qi Dai, Alexander G. Hauptmann, Zhi-Qi Cheng• 2025

Related benchmarks

TaskDatasetResultRank
Goal Conditioned Visual NavigationGo Stanford (evaluation)
Success Rate (SR)75
8
Goal Conditioned Visual NavigationReCon (evaluation)
Success Rate (SR)93
8
Goal Conditioned Visual NavigationSCAND (evaluation)
Success Rate (SR)68
8
Goal Conditioned Visual NavigationHuRoN (evaluation)
Success Rate76
8
Humanoid Navigation1X Humanoid Dataset navigation 2024
Success Rate81
8
NavigationTartanDrive (unseen)
Success Rate (SR)42
8
NavigationGo Stanford, ReCon, SCAND, HuRoN Average
Average Inference Time (s)16
5
Visual Quality AssessmentGo Stanford, ReCon, SCAND, and HuRoN (evaluation)
SSIM0.457
4
Showing 8 of 8 rows

Other info

Follow for update