Interactive Serving on Llama 3.1-8B-Instruct (512->128 tokens, concurrency=1)
269Throughput (Tok/s)
Pre-comp. clustering rebuild
Updated 1mo ago
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
2026.03 | 269 | 17 | 3.61 | 186 | 0.87 | 3.1 | |
2026.03 | 259 | 18.6 | 3.74 | 188 | 0.83 | 6.1 | |
2026.03 | 157 | 25.6 | 6.21 | 232 | 0.51 | 6.9 | |
2026.03 | 155 | 25.1 | 6.3 | 266 | 0.5 | 15 | |
2026.03 | 34 | 135 | 28.2 | 166 | 0.11 | 3 | |
2026.03 | 14 | 373 | 67 | 177 | 0.05 | 6.5 |