Scene and Human in One World: Reconstruction in a Feedforward Pass
About
Reconstructing humans in dynamic scenes from moving monocular cameras remains challenging due to scale ambiguity, human-scene misalignment, and occlusion interference. Rather than treating human mesh recovery and scene reconstruction as separate tasks, we believe that accurate human-scene reconstruction requires the two tasks to mutually inform each other: parametric human models offer semantic structure and metric-scale priors, while scene geometry provides spatial context for human localization and alignment. Built on this insight, we introduce SHOW, a mask-promptable human mesh recovery framework that couples feed-forward 3D scene reconstruction with Human Mesh Recovery in a unified metric space. SHOW injects human semantics and scale priors from parametric human models into normalized point-map prediction, enabling metric-scale scene reconstruction from inherently scale-ambiguous monocular input. In turn, the recovered scene geometry constrains human mesh estimation, encouraging spatially consistent human placement and improved human-scene alignment. To handle complex multi-person and cluttered scenes, SHOW further incorporates a promptable masking mechanism that enables flexible target-human selection while suppressing background distractions and occlusion interference. Through joint training, the model learns both human-aware geometric features and geometry-constrained human features, producing aligned metric-scale reconstructions from monocular human-centric videos. Extensive experiments demonstrate that SHOW improves metric-scale consistency, human-scene alignment, and reconstruction accuracy under challenging camera motion, occlusion, and cluttered backgrounds.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| 3D Human Mesh Recovery | 3DPW | PA-MPJPE41 | 89 | |
| Human Mesh Recovery | EMDB 1 | PA-MPJPE45.8 | 21 | |
| Global human motion estimation | RICH (test) | WA-MPJPE107.3 | 15 | |
| Global human motion estimation | EMDB-2 (subset 2) | WA-MPJPE109.1 | 10 | |
| Human-scene consistency | 3DPW | HS-V50.03 | 3 | |
| Human-scene consistency | EMDB 1 | HS-V5 Consistency Score0.013 | 3 |