Share your thoughts, 1 month free Claude Pro on usSee more

P99 Per-Token Latency on LLaMA-2 7B (inference)

8.23P99 Per-Token Latency (ms)

JIT+CUDA

Updated 1mo ago

Evaluation Results

Method	Links
JIT+CUDA 2026.04		8.23
JIT+CUDA 2026.04		9.44
JIT+CUDA 2026.04		11.36
JIT+CUDA 2026.04		12.54
JIT+CUDA 2026.04		14.11
JIT+CUDA 2026.04		15.65
TensorRT–LLM 2026.04		16.53
TensorRT–LLM 2026.04		16.54
TensorRT–LLM 2026.04		16.54
TensorRT–LLM 2026.04		16.56
TensorRT–LLM 2026.04		16.61
TensorRT–LLM 2026.04		16.65
TensorRT–LLM 2026.04		16.73
JIT+CUDA 2026.04		17.31
HuggingFace 2026.04		18.3
HuggingFace 2026.04		18.38
HuggingFace 2026.04		18.39
HuggingFace 2026.04		18.39
HuggingFace 2026.04		18.41
HuggingFace 2026.04		18.42
HuggingFace 2026.04		18.44
HuggingFace 2026.04		18.46
HuggingFace 2026.04		18.49
HuggingFace 2026.04		18.49
JIT+CUDA 2026.04		18.9
HuggingFace 2026.04		18.94
JIT+CUDA 2026.04		20.5
JIT+CUDA 2026.04		22.09
JIT+CUDA 2026.04		23.68
TensorRT–LLM 2026.04		24.03
TensorRT–LLM 2026.04		31.59
TensorRT–LLM 2026.04		52.28
TensorRT–LLM 2026.04		68.84