Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views

About

Though recent advances in vision-language models (VLMs) have achieved remarkable progress across a wide range of multimodal tasks, understanding 3D spatial relationships from limited views remains a significant challenge. Previous reasoning methods typically rely on pure text (e.g., topological cognitive maps) or on 2D visual cues. However, their limited representational capacity hinders performance in specific tasks that require 3D spatial imagination. To address this limitation, we propose 3DThinker, a framework that can effectively exploits the rich geometric information embedded within images while reasoning, like humans do. Our framework is the first to enable 3D mentaling during reasoning without any 3D prior input, and it does not rely on explicitly labeled 3D data for training. Specifically, our training consists of two stages. First, we perform supervised training to align the 3D latent generated by VLM while reasoning with that of a 3D foundation model (e.g., VGGT). Then, we optimize the entire reasoning trajectory solely based on outcome signals, thereby refining the underlying 3D mentaling. Extensive experiments across multiple benchmarks show that 3DThinker consistently outperforms strong baselines and offers a new perspective toward unifying 3D representations into multimodal reasoning. Our code is available at https://github.com/zhangquanchen/3DThinker.

Zhangquan Chen, Manyuan Zhang, Xinlei Yu, Xufang Luo, Mingze Sun, Zihao Pan, Xiang An, Yan Feng, Peng Pei, Xunliang Cai, Ruqi Huang• 2025

Related benchmarks

TaskDatasetResultRank
Spatial ReasoningEgo3D-Bench
Ego Distance Score69.9
52
Spatial UnderstandingMindCube (tiny)
Rotation Score57
37
Spatial ReasoningMindCube tiny (test)
Rot. Accuracy55
30
3D Spatial Reasoning3DSRBench
Accuracy65.6
27
Spatial ReasoningVSI-Bench MV 76
Accuracy63.7
11
Spatial ReasoningCV-Bench SI 58
Accuracy81.1
11
Spatial ReasoningSPAR-Bench SI, MV 91
Accuracy63.3
11
Spatial ReasoningViewSpatial-Bench SI, MV 34
Accuracy68.6
11
Spatial ReasoningMMSI-Bench 79 (MV)
Accuracy43.3
11
Spatial ReasoningSPBench SI, MV 35
Accuracy68.3
11
Showing 10 of 10 rows

Other info

Follow for update