Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

World Tracing: Generative Pixel-Aligned Geometry Beyond the Visible

About

Image-to-3D methods often trade off faithfulness and completeness: depth estimators are anchored to input pixels but stop at the visible surface, while image-to-3D models generate complete shapes that are often misaligned with the input. We introduce World Tracing, a generative pixel-aligned geometry representation that predicts 3D points aligned with observed pixels while completing geometry beyond the visible surface. For each input pixel, World Tracing predicts an ordered stack of camera-space 3D points, where the first layer represents the visible surface and subsequent layers represent front-to-back intersections with occluded surfaces. We instantiate this representation with a world-tracing diffusion transformer, WT-DiT, which treats multiple geometry layers as separate denoising tokens coupled through factorized and global attention. WT-DiT is trained with pixel-space flow matching and a mixed noise schedule that balances visible-surface reconstruction with occluded-geometry generation. World Tracing achieves strong performance on visible-surface reconstruction and complete geometry generation across object, scene, and dynamic benchmarks, outperforming both depth predictors and image-to-3D generators. It also preserves 2D-to-3D correspondence, enabling text-driven 3D scene editing, geometry-conditioned novel-view video synthesis, and training-free integration with textured-mesh generators.

Hao Zhang, Mohamed El Banani, Jen-Hao Cheng, Paul Zhang, Yi Hua, Ben Mildenhall, Christoph Lassner, Narendra Ahuja, Gengshan Yang• 2026

Related benchmarks

TaskDatasetResultRank
Depth EstimationNYU Depth V2
RMSE0.1312
226
Depth EstimationETH3D Indoor
AbsRel0.0332
8
Depth EstimationETH3D Outdoor
AbsRel0.0451
8
Scene Geometry3D-FRONT 50 samples (held-out)
MAE0.0102
7
Image-to-3D Reconstruction100-sample object benchmark
L1 Error0.0213
6
Scene GeometryInternal generalization probe (test)
MAE0.0328
6
Visible Surface Depth EstimationObjects 100-sample benchmark (held-out)
MAE0.0149
6
Dynamic Geometry ReconstructionTruebone
Global CD-L20.0063
4
Dynamic Geometry ReconstructionObjaverse-XL (val)
Global CD-L20.0034
4
Dynamic Geometry ReconstructionActionBench
Global CD-L20.0291
4
Showing 10 of 10 rows

Other info

GitHub

Follow for update