Evaluation Suite
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Zero-shot Reasoning | Evaluation Suite Zero-shot (OpenbookQA, ARC-e, ARC-c, WinoGrande, HellaSwag, PIQA, MathQA) | Average Accuracy69.99 | 97 | |
| Classification | Zero-shot Evaluation Suite (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA) | Average Accuracy (Zero-Shot Suite)69.99 | 94 | |
| Zero-shot Evaluation | Zero-shot Evaluation Suite (ARC, Hellaswag, LAMBADA, PIQA, Winogrande) | ARC-C73.39 | 85 | |
| Zero-shot Evaluation | Zero-shot Evaluation Suite (ARC-c, ARC-e, HellaSwag, PIQA, WinoGrande) | ARC-c Accuracy59.3 | 82 | |
| Question Answering | Evaluation Suite (ARC, HellaSwag, MMLU) Zero-shot (test) | ARC-C52.9 | 67 | |
| Zero-shot Reasoning and General Knowledge | Evaluation Suite Zero-shot (PIQA, ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, MathQA, MMLU) | PIQA (PQ) Accuracy83.68 | 40 | |
| Image Classification | Evaluation suite OOD | Average Accuracy68.3 | 36 | |
| Image Classification | Evaluation suite (ID) | Average Accuracy93 | 36 | |
| Image Classification | Zero-shot Evaluation Suite (Food-101, CIFAR-10, CIFAR-100, SUN397, Stanford Cars, FGVC Aircraft, DTD, Oxford-IIIT Pets, Caltech-101, Flowers102, ImageNet-1K) various (test) | Food-101 Top-1 Acc90.5 | 29 | |
| Zero-shot Evaluation | Zero-shot Evaluation Suite | Average Score77.28 | 29 | |
| Zero-shot Language Understanding | Evaluation Suite Zero-shot (LMB, HellA, PIQA, ARC-E, ARC-C, WINO, Open, MMLU) | ARC-E Accuracy83.4 | 25 | |
| Language Understanding and Common Sense Reasoning | Zero-shot Evaluation Suite (PIQA, ARC-C, ARC-E, HellaS, WinoG, BoolQ, LAMBADA, C4) | PIQA Accuracy79.67 | 24 | |
| Zero-shot Question Answering and Reasoning | Evaluation Suite Zero-shot (ARC, LogiQA, Wino, CSQA, BoolQ, PIQA, MMLU) | ARC83.88 | 21 | |
| Multi-task Language Evaluation | 16-task evaluation suite 1B/25B checkpoints (test) | Average Score42.22 | 20 | |
| General Language Understanding | 12-task evaluation suite (test) | Average Score71.62 | 20 | |
| Aggregate Multimodal Evaluation | Evaluation Suite Average | Average Score100 | 19 | |
| Question Answering and Commonsense Reasoning | Zero-Shot Evaluation Suite (Arc-e, Arc-c, Boolq, Hellaswag, Openbookqa, Piqa, SciQ, Winogrande) | ARC-E65.74 | 18 | |
| Commonsense Reasoning | Zero-shot Evaluation Suite (HellaSwag, PIQA, Arc-E, Arc-C, WinoGrande, OBQA, SIQA, BoolQ) | HellaSwag (Zero-shot)38.29 | 15 | |
| Large Language Model Evaluation | 12-task evaluation suite composite (test) | Reading Comprehension Score49.6 | 14 | |
| Language Understanding | Evaluation suite (Commonsense, Comprehension, Knowledge, Reasoning, Language tasks) (test) | Average Accuracy43.8 | 12 | |
| Zero-shot Classification | Evaluation Suite Zero-shot (PiQA, LAMBDA, ARC-e, ARC-c, HellaS) | Decode Latency5.23 | 12 | |
| Natural Language Understanding | Evaluation Suite Zero-shot (Arc-c, Arc-e, BoolQ, COPA, MMLU, OBQA, PIQA, RTE, Winogrande) | ARC-c56.14 | 12 | |
| Zero-shot Classification | Zero-shot Evaluation Suite (AC, AE, WI, QA) v1 | AC Score46.2 | 10 | |
| Question Answering | Evaluation Suite Zero-shot | Accuracy (Zero-shot)54.3 | 9 | |
| Zero-shot Evaluation | Zero-shot Evaluation Suite Aggregate | Average Accuracy (Zero-shot)73 | 7 |
Showing 25 of 33 rows