Share your thoughts, 1 month free Claude Pro on usSee more

LLM Serving on High-load generation workload w/o NVLink

878Time To First Token (ms)

DeInfer (w/ low-rank KV cache)

Updated 1mo ago

Evaluation Results

Method	Links
DeInfer (w/ low-rank KV cache) 2026.04		878	154
DeInfer (w/ low-rank KV cache) 2026.04		906	101
DeInfer 2026.04		927	47
DeInfer (w/ low-rank KV cache) 2026.04		940	283
DeInfer 2026.04		944	46
DeInfer 2026.04		960	57
DeInfer 2026.04		1,812	88
DeInfer (w/ low-rank KV cache) 2026.04		1,840	354
DeInfer (w/ low-rank KV cache) 2026.04		2,085	216
DeInfer 2026.04		2,088	91
DeInfer (w/ low-rank KV cache) 2026.04		2,093	105
DeInfer (w/ low-rank KV cache) 2026.04		2,102	115
DeInfer 2026.04		2,110	88
DeInfer 2026.04		2,137	108
DeInfer 2026.04		2,192	115
DeInfer (w/ low-rank KV cache) 2026.04		2,216	660
DeInfer 2026.04		2,246	87
DeInfer (w/ low-rank KV cache) 2026.04		2,484	143
Base 2026.04		3,126	54
Base 2026.04		3,566	47
Base 2026.04		3,845	47
Base 2026.04		8,164	206
Base 2026.04		8,950	156
Base 2026.04		9,359	103
Base 2026.04		10,064	122
Base 2026.04		10,451	101
Base 2026.04		10,764	96