AIR-Bench
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Instruction Following | Air-Bench | Score90 | 18 | |
| Sound Foundation | AIR-Bench 1.0 (test) | Score65.1 | 13 | |
| Safety | Air-Bench 2024 | Safety Score91.7 | 12 | |
| Safety | AIR-Bench | Average Score0.66 | 12 | |
| Paralinguistic speech understanding | AIR-Bench Speech (test) | Emotion Acc71.45 | 11 | |
| Chat Benchmark | AIR-Bench | Score (Speech Domain)7.54 | 11 | |
| Speech Understanding | AIR-Bench | SER29.9 | 10 | |
| Retrieval | AIR-Bench English 24.04 | Wiki Score65.5 | 10 | |
| Audio Perception | AIR-Bench Foundation | Speech Performance63.2 | 9 | |
| Open-formed Audio Question Answering | AIR-Bench Music | Score6.16 | 8 | |
| Audio Classification | AIR-Bench Speech | Emotion Acc (MELD)47.16 | 8 | |
| Open-formed Audio Question Answering | AIR-Bench Sound | Score7.01 | 8 | |
| Question Answering | AIR-Bench Foundation | Accuracy36.8 | 8 | |
| Content Moderation | AIR-Bench Text + Image (test) | Precision83 | 8 | |
| Content Moderation | AIR-Bench Image Only (test) | Precision94 | 8 | |
| Content Moderation | AIR-Bench Text Only (test) | Precision94 | 8 | |
| Music Foundation Tasks | AIR-Bench Music 1.0 (test) | Inst. Classification Acc65.8 | 7 | |
| Speech Foundation | AIR-Bench Speech Foundation | Speech Grounding5,920 | 7 | |
| Speech Chat | AIR-Bench 1.0 (test) | Overall Score7.18 | 7 | |
| Safety Evaluation | AIR-Bench 64 | Harm Score3.12 | 6 | |
| Audio Understanding | AIR-Bench | Overall Score66.25 | 6 | |
| Gender Classification | Air-Bench | Accuracy0.905 | 6 | |
| Retrieval | AIR-Bench | Spearman's Rho0.71 | 5 | |
| Safety Alignment | AIR-Bench 64 harmful prompts | Harm1.13 | 4 | |
| Auditory Understanding | AIR-Bench foundation | Accuracy64.85 | 4 |
Showing 25 of 27 rows