LLM Generation Performance on A100 80GB (inference)
128Maximum Batch Size
Layer-Condensed KV Cache
Updated 1mo ago
Evaluation Results
| Method | Links | ||
|---|---|---|---|
2024.05 | 128 | 421.02 | |
2024.05 | 42 | 315.09 | |
2024.05 | 32 | 108.29 | |
2024.05 | 15 | 141.1 | |
2024.05 | 8 | 77.65 | |
2024.05 | 1 | 14.1 |