Share your thoughts, 1 month free Claude Pro on usSee more

LLM Generation on RTX 3090 24GB (inference)

1,150Max Batch Size

Layer-Condensed KV Cache

Updated 1mo ago

Evaluation Results

Method	Links
Layer-Condensed KV Cache 2024.05		1,150	10,033.4
Layer-Condensed KV Cache 2024.05		384	4,113.37
Layer-Condensed KV Cache 2024.05		289	7,239.92
Llama 2024.05		239	5,142.86
Layer-Condensed KV Cache 2024.05		119	2,374.05
Layer-Condensed KV Cache 2024.05		95	378.89
Layer-Condensed KV Cache 2024.05		83	5.99
Layer-Condensed KV Cache 2024.05		72	401.92
Layer-Condensed KV Cache 2024.05		64	534.02
Llama 2024.05		48	1,424.96
Layer-Condensed KV Cache 2024.05		32	380.6
Layer-Condensed KV Cache 2024.05		23	171.65
Layer-Condensed KV Cache 2024.05		23	1.63
Layer-Condensed KV Cache 2024.05		16	315.38
Layer-Condensed KV Cache 2024.05		16	310.05
Layer-Condensed KV Cache 2024.05		12	151.91
Llama 2024.05		9	225.31
Layer-Condensed KV Cache 2024.05		8	119.68
Llama 2024.05		7	174.11
Llama 2024.05		5	140.88
Layer-Condensed KV Cache 2024.05		4	83.8
Llama 2024.05		4	0.23
Llama 2024.05		2	56.98
Llama 2024.05		1	32.02