Spec-Bench
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Speculative Decoding | Spec-Bench | MT Score195.6 | 57 | |
| Inference Acceleration | Spec-Bench | Speedup5.31 | 53 | |
| Text Generation | Spec-Bench Overall | SD Score2.33 | 21 | |
| Translation | Spec-Bench Trans. | CR6.41 | 21 | |
| Summarization | Spec-Bench Sum. | CR Score4.73 | 21 | |
| Retrieval-Augmented Generation | Spec-Bench RAG | CR5.48 | 21 | |
| Question Answering | Spec-Bench QA | CR4.54 | 21 | |
| Multi-turn Dialogue | Spec-Bench Multi. | CR3.22 | 21 | |
| Mathematical Reasoning | Spec-Bench Math | CR4.09 | 21 | |
| Language Model Decoding | Spec-Bench | Conv. Acc267.6 | 11 | |
| Speculative Decoding Throughput | Spec-Bench | Throughput (Conv.)519.7 | 10 | |
| Speculative Decoding | Spec-Bench OLMo 2 7B | Conversation Score5.12 | 5 | |
| Speculative Decoding | Spec-Bench Llama2-7B v1.0 (test) | MT Score2.73 | 4 | |
| Speculative Decoding Throughput | Spec-Bench (test) | Throughput (Conv.)- | 0 |
Showing 14 of 14 rows