FoundationStereo: Zero-Shot Stereo Matching
About
Tremendous progress has been made in deep stereo matching to excel on benchmark datasets through per-domain fine-tuning. However, achieving strong zero-shot generalization - a hallmark of foundation models in other computer vision tasks - remains challenging for stereo matching. We introduce FoundationStereo, a foundation model for stereo depth estimation designed to achieve strong zero-shot generalization. To this end, we first construct a large-scale (1M stereo pairs) synthetic training dataset featuring large diversity and high photorealism, followed by an automatic self-curation pipeline to remove ambiguous samples. We then design a number of network architecture components to enhance scalability, including a side-tuning feature backbone that adapts rich monocular priors from vision foundation models to mitigate the sim-to-real gap, and long-range context reasoning for effective cost volume filtering. Together, these components lead to strong robustness and accuracy across domains, establishing a new standard in zero-shot stereo depth estimation. Project page: https://nvlabs.github.io/FoundationStereo/
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Stereo Matching | KITTI 2012 (test) | -- | 76 | |
| Stereo Matching | ETH3D | Threshold Error > 1px (All)0.48 | 30 | |
| Stereo Matching | Booster Q (test) | Error Rate (> 2%)5.18 | 26 | |
| Stereo Depth Estimation | SQUID zero-shot | Relative Error (Rel)0.1095 | 16 | |
| Stereo Matching | DrivingStereo Zero-shot generalization | Error Rate (Sunny)3.22 | 15 | |
| Stereo Depth Estimation | TartanAir underwater (test) | Relative Error (Rel)0.0542 | 13 | |
| 3D Reconstruction | ShapeR Evaluation Dataset 1.0 (test) | CD6.483 | 10 | |
| Stereo Depth Estimation | SUDS On Geometry Only | EPE1.89 | 8 | |
| DSM Reconstruction | Omaha Diachronic DFC2019 | Altitude MAE1.3 | 8 | |
| DSM Reconstruction | Jacksonville DFC2019 | Altitude MAE (m)2.33 | 8 |