Share your thoughts, 1 month free Claude Pro on usSee more

LLM Serving on High-load generation workload w/ NVLink

443TTFT (ms)

DeInfer

Updated 1mo ago

Evaluation Results

Method	Links
DeInfer 2026.04		443	33
DeInfer 2026.04		448	37
DeInfer (w/ low-rank KV cache) 2026.04		457	86
DeInfer (w/ low-rank KV cache) 2026.04		514	146
DeInfer 2026.04		581	50
DeInfer (w/ low-rank KV cache) 2026.04		640	276
Base 2026.04		649	41
Base 2026.04		654	42
Base 2026.04		724	50
DeInfer 2026.04		749	56
DeInfer (w/ low-rank KV cache) 2026.04		782	184
DeInfer 2026.04		797	78
DeInfer (w/ low-rank KV cache) 2026.04		817	93
DeInfer 2026.04		875	64
DeInfer (w/ low-rank KV cache) 2026.04		901	330
DeInfer (w/ low-rank KV cache) 2026.04		948	96
DeInfer 2026.04		983	76
DeInfer 2026.04		1,162	92
DeInfer (w/ low-rank KV cache) 2026.04		1,214	642
DeInfer (w/ low-rank KV cache) 2026.04		1,377	127
Base 2026.04		1,417	98
Base 2026.04		1,461	94
DeInfer 2026.04		1,463	90
Base 2026.04		1,499	115
Base 2026.04		1,525	191
Base 2026.04		1,605	93
Base 2026.04		1,662	156