Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

STORM: Spatio-Temporal Reconstruction Model for Large-Scale Outdoor Scenes

About

We present STORM, a spatio-temporal reconstruction model designed for reconstructing dynamic outdoor scenes from sparse observations. Existing dynamic reconstruction methods often rely on per-scene optimization, dense observations across space and time, and strong motion supervision, resulting in lengthy optimization times, limited generalization to novel views or scenes, and degenerated quality caused by noisy pseudo-labels for dynamics. To address these challenges, STORM leverages a data-driven Transformer architecture that directly infers dynamic 3D scene representations--parameterized by 3D Gaussians and their velocities--in a single forward pass. Our key design is to aggregate 3D Gaussians from all frames using self-supervised scene flows, transforming them to the target timestep to enable complete (i.e., "amodal") reconstructions from arbitrary viewpoints at any moment in time. As an emergent property, STORM automatically captures dynamic instances and generates high-quality masks using only reconstruction losses. Extensive experiments on public datasets show that STORM achieves precise dynamic scene reconstruction, surpassing state-of-the-art per-scene optimization methods (+4.3 to 6.6 PSNR) and existing feed-forward approaches (+2.1 to 4.7 PSNR) in dynamic regions. STORM reconstructs large-scale outdoor scenes in 200ms, supports real-time rendering, and outperforms competitors in scene flow estimation, improving 3D EPE by 0.422m and Acc5 by 28.02%. Beyond reconstruction, we showcase four additional applications of our model, illustrating the potential of self-supervised learning for broader dynamic scene understanding.

Jiawei Yang, Jiahui Huang, Yuxiao Chen, Yan Wang, Boyi Li, Yurong You, Apoorva Sharma, Maximilian Igl, Peter Karkus, Danfei Xu, Boris Ivanovic, Yue Wang, Marco Pavone• 2024

Related benchmarks

TaskDatasetResultRank
New View SynthesisWaymo (val)
PSNR (dB)23.76
14
Sparse depth estimationWaymo Open Dataset
RMSE (Dynamic Only)7.5
8
Dynamic ReconstructionWaymo Open Dataset (WOD) Dynamic-only regions (val)
D-RMSE7.5
7
Dynamic ReconstructionWaymo Open Dataset (WOD) Full image, non-sky (val)
D-RMSE5.47
7
Novel View SynthesisWaymo Open Dataset original-view dynamic-only (test)
PSNR22.1
6
Novel View SynthesisWaymo Open Dataset original-view full-image (test)
PSNR26.38
6
4D Scene ReconstructionWaymo Dynamic-only (val)
PSNR21.26
5
Scene Flow EstimationWaymo Open Dataset (WOD)
EPE (m)0.304
5
Novel View SynthesisKITTI In-Domain (val)
PSNR18.96
5
Novel View SynthesisPandaSet Out-of-Domain (val)
PSNR23.13
5
Showing 10 of 15 rows

Other info

Follow for update