VideoMME
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Video Understanding | VideoMME | Overall Score100 | 192 | |
| Video Understanding | VideoMME | Score (Short)77 | 127 | |
| Video Question Answering | VideoMME | Accuracy85.1 | 99 | |
| Video Question-Answering | VideoMME wo sub | Accuracy88.6 | 51 | |
| Video Question Answering | VideoMME 16 (test) | Medium Length Score70.11 | 45 | |
| Multi-modal Video Evaluation | VideoMME | Score (No Subscript)75 | 30 | |
| Video Understanding | VideoMME Long | Score59 | 25 | |
| Video Understanding | VideoMME v1.0 (test) | Score60.8 | 21 | |
| Video Understanding | VideoMME | Wall-time Speedup3.27 | 21 | |
| Long Video Understanding | VideoMME | Accuracy81.3 | 21 | |
| Video Question Answering | VideoMME KFS-Bench full set | Accuracy68.4 | 20 | |
| Video Question Answering | VideoMME (long split) | Accuracy67.4 | 18 | |
| Video Question Answering | VideoMME | VQA Score (wo subs)66.3 | 17 | |
| Long Video Understanding | VideoMME Long split, 30-60 min | Accuracy65.3 | 15 | |
| Video Question Answering | VideoMME w/ sub | Score87.8 | 15 | |
| Video Question Answering | VideoMME with subtitles | Acc (Overall)64.8 | 15 | |
| Multiple-Choice Video QA | VideoMME | Accuracy75 | 15 | |
| Video understanding | VideoMME wo-subs | Accuracy71.9 | 13 | |
| Video Question Answering | VideoMME 2024 (test) | Accuracy (Short, w/o Subtitles)81.7 | 13 | |
| Multi-choice Video Question Answering | VideoMME | Accuracy53.4 | 13 | |
| Long-context video understanding | VideoMME | Accuracy (w subs)81.3 | 13 | |
| Video Understanding | VideoMME Overall | Performance (No Subtitles)75 | 9 | |
| Video Question Answering | VideoMME Overall 1.0 (test) | Accuracy68.3 | 8 | |
| Video Question Answering | VideoMME Medium 1.0 (test) | Accuracy66.6 | 8 | |
| Video Question Answering | VideoMME w/o sub. 1.0 (test) | Overall Acc65 | 8 |
Showing 25 of 38 rows