Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale
About
Affordance grounding aims to localize where to interact with an object, a fundamental capability for embodied agents. Yet progress is bottlenecked by data: manual annotation is prohibitively expensive and confines existing datasets to a narrow set of predefined object and affordance categories. We introduce Affogato, a framework for open-vocabulary affordance grounding centered on Affogato-750K, a large-scale dataset of 750K 3D affordance heatmaps paired with natural language queries. We build it with a fully automated pipeline that orchestrates foundation models to generate them at scale without human labeling. It covers significantly more diverse categories than any existing dataset. For reliable evaluation, we further provide 5K human-verified test pairs. We also present Espresso-3D and Espresso-2D, simple yet effective models with a unified architecture across both modalities. Pretraining on Affogato-750K improves both Espresso and prior methods and yields the largest gains on unseen object and affordance categories, showing that it provides broadly transferable supervision across architectures.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Affordance Grounding | AGD20k v1 (Seen) | KLD1.426 | 28 | |
| Affordance Grounding | AGD20k v1 (Unseen) | KLD1.571 | 28 | |
| Affordance prediction | AGD20K unseen | KLD1.034 | 26 | |
| 2D affordance grounding | AGD20K unseen (test) | KLD0.974 | 14 | |
| 3D Affordance Grounding | Affogato (test) | aIoU13.6 | 14 | |
| Open-vocabulary 3D affordance grounding | LASO seen (test) | aIoU21.9 | 10 | |
| Open-vocabulary 3D affordance grounding | LASO unseen (test) | aIoU20.8 | 10 | |
| 3D Affordance Grounding | Affogato-750K All -> All | aIoU13.6 | 3 | |
| 3D Affordance Grounding | Affogato-750K Daily-used -> Furnitures | aIoU (Affogato-750K Furnitures)18.2 | 3 | |
| 3D Affordance Grounding | Affogato-750K Furnitures -> Daily-used | aIoU4.6 | 3 |