| Dataset Name | SOTA Method | Metric | Trend | ||
|---|---|---|---|---|---|
| High-load generation workload w/ NVLink | DeInfer | TTFT (ms)443 | 27 | 3mo ago | |
| High-load generation workload w/o NVLink | DeInfer (w/ low-rank KV cache) | Time To First Token (ms)878 | 27 | 3mo ago | |
| Evol-CodeAlpaca | Average Latency (ms/token)154.52 | 12 | 26d ago | ||
| MATH-plus | Average Latency (ms/token)107.1 | 12 | 26d ago | ||
| Alpaca | Average Latency (ms/token)111.46 | 12 | 26d ago | ||
| Evol-CodeAlpaca | p90 Latency (ms/token)234.09 | 12 | 26d ago | ||
| MATH-plus | p90 Latency (ms/token)157.83 | 12 | 26d ago | ||
| AGENTPREFIX derived from τ-bench (trace) | ContextPilot | TTFT P50 (s)21.39 | 12 | 2mo ago | |
| ToolBench | AugServe | Effective Throughput (req/s)1.11 | 9 | 4mo ago | |
| Merge dataset | AugServe | Effective Throughput (req/s)0.6 | 9 | 4mo ago | |
| Synthetic workloads (maximum serving capacity) | DuetServe | Throughput (req/s)12.8 | 6 | 1mo ago | |
| LLaMA-2 70B chatbot workload | GPU-Only | TTFT (ms)45.2 | 4 | 4mo ago | |
| Real-world trace (ShareGPT and BurstGPT) | DiLaServe | SLO Attainment91.13 | 3 | 25d ago | |
| Azure-Conv | DUETSERVE | Throughput (req/s)8.02 | 2 | 1mo ago | |
| Node multi-node inference workload | PALS | Efficiency Gain26.3 | 1 | 2mo ago |
