LLM Serving on LLaMA-2 70B chatbot workload
45.2TTFT (ms)
GPU-Only
Updated 1mo ago
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
2025.12 | 45.2 | 18.3 | 19.4 | 21.2 | |
2025.12 | 46.8 | 23.5 | 27.3 | 31.6 | |
2025.12 | 47.1 | 19.8 | 21.7 | 23.8 | |
2025.12 | 52.7 | 28.6 | 35.4 | 42.8 |
GPU-Only
| Method | Links | ||||
|---|---|---|---|---|---|
2025.12 | 45.2 | 18.3 | 19.4 | 21.2 | |
2025.12 | 46.8 | 23.5 | 27.3 | 31.6 | |
2025.12 | 47.1 | 19.8 | 21.7 | 23.8 | |
2025.12 | 52.7 | 28.6 | 35.4 | 42.8 |