OpenEQA
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Embodied Question Answering | OpenEQA v1.0 (test) | LLM-Match67.7 | 32 | |
| Episodic-memory Question Answering | OpenEQA v1 (ScanNet) | LLM-Match87.7 | 29 | |
| Embodied Question Answering | OpenEQA | Accuracy69.3 | 26 | |
| Embodied Question Answering | OpenEQA EM-EQA | Accuracy86.8 | 24 | |
| Open-ended Video Reasoning | OpenEQA (test) | Accuracy (< 30s)58.6 | 18 | |
| 3D Question Answering | OpenEQA | LLM-Match56.2 | 12 | |
| Active Question Answering | OpenEQA v1 (HM3D) | LLM-Match85.1 | 12 | |
| Episodic-memory Question Answering | OpenEQA v1 (HM3D) | LLM Match Score85.1 | 12 | |
| Embodied Question Answering | OpenEQA EM-EQA Episodes up to 32 frames | LLM-Match Score86.8 | 10 | |
| Zero-shot understanding of embodied scenes | OpenEQA | Score51.2 | 10 | |
| Embodied Question Answering | OpenEQA (test) | GPT-4 Score86.8 | 10 | |
| Embodied Question Answering | OpenEQA EM-EQA | LLM-Match86.8 | 8 | |
| Episodic Memory Embodied Question Answering | OpenEQA 1/10 subset Unseen (val) | LLM Match Score61.5 | 7 | |
| Embodied Question Answering | OpenEQA v1 (test) | Score85.1 | 5 | |
| Active Embodied Question Answering | OpenEQA 184 | LLM-SR52.6 | 4 |
Showing 15 of 15 rows