SLURP
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Selective capability unlearning | SLURP Retain Set 1.0 (DR) | IF90.18 | 40 | |
| Slot Filling | SLURP (test) | F1 Score80.61 | 26 | |
| Intent Classification | SLURP (test) | Accuracy (IC)90.07 | 26 | |
| Selective capability unlearning | SLURP Forget Set (DF) 1.0 | IF95.27 | 20 | |
| Intent Detection | SLURP | Accuracy89.16 | 16 | |
| ASR Rescoring | SLURP (test) | WER23.77 | 11 | |
| Spoken Language Understanding | SLURP 6 tasks | Avg Accuracy84.41 | 9 | |
| Spoken Language Understanding | SLURP 3 tasks | Average Accuracy84.41 | 9 | |
| Spoken Language Understanding | SLURP Oracle | Intent Accuracy91.57 | 8 | |
| Slot Filling | SLURP Reference transcriptions (test) | SLU F1 Score86.3 | 6 | |
| Slot Filling | SLURP Whisper Medium transcriptions (test) | SLU F1 Score66.5 | 6 | |
| Slot Filling | SLURP Whisper Small transcriptions (test) | SLU-F164.6 | 6 | |
| Slot Filling | SLURP Whisper Base transcriptions WER 26.4% (test) | SLU F1 Score59.1 | 6 | |
| Slot Filling | SLURP Whisper Tiny transcriptions (test) | SLU F1 Score56.4 | 6 | |
| Spoken Language Understanding | SLURP (test) | Scenario Accuracy78.76 | 5 | |
| Spoken Language Understanding | SLURP 0 dB | Intent Accuracy33.29 | 4 | |
| Spoken Language Understanding | SLURP 2.5 dB | Intent Accuracy46.68 | 4 | |
| Spoken Language Understanding | SLURP 5 dB | Intent Accuracy59.47 | 4 | |
| Spoken Language Understanding | SLURP 10 dB | Intent Accuracy74.94 | 4 | |
| Spoken Language Understanding | SLURP Clean | Intent Accuracy86.02 | 4 | |
| Speech Intent Classification and Slot Filling | SLURP (test) | Intent Accuracy90.68 | 4 |
Showing 21 of 21 rows