Share your thoughts, 1 month free Claude Pro on usSee more

Long-context evaluation on RULER 64k

100VT Score

Llama-3.1-8B

Updated 15d ago

Evaluation Results

Method
Llama-3.1-8B 2025.05	100	100	100	100	100	100	96	99	100	14.8	92	60	52	85.6	-	-	-	-	-	-	-
Dense 2025.03	97.29	100	100	-	98.96	97.92	-	97.66	98.96	-	85.42	83.33	59.38	91.89	-	-	-	-	-	-	-
Full Attn 2026.05	96.8	-	-	-	-	-	-	-	-	65.28	84	-	-	86.23	100	-	63.4	69.83	99.5	97.6	99.66
SnapKV 2025.03	95	100	100	-	98.96	94.79	-	97.66	100	-	68.75	83.33	58.33	89.68	-	-	-	-	-	-	-
Quest 2026.05	94.8	-	-	-	-	-	-	-	-	36.56	63.8	-	-	70.6	87.93	-	54.6	62.57	80.4	75.9	78.86
RTPurbo 2026.05	94.6	-	-	-	-	-	-	-	-	65.1	81.4	-	-	85.49	99.93	-	62.6	70.03	99.65	97.5	98.6
Qwen3-4B-SFT w/ LongR 2026.02	93.32	-	-	-	-	-	-	-	-	-	-	-	-	69.43	71.36	43.6	-	-	-	-	-
FlexPrefill 2026.05	92.72	-	-	-	-	-	-	-	-	51.04	80.07	-	-	77.77	100	-	58	68.63	96.35	55.5	97.6
Qwen3-4B-SFT w/ DAPO 2026.02	91.96	-	-	-	-	-	-	-	-	-	-	-	-	63.53	67.74	30.9	-	-	-	-	-
RazorAttn 2026.05	91.52	-	-	-	-	-	-	-	-	65.96	83.8	-	-	85.11	99.73	-	62.8	69.33	98.95	95.2	98.73
TokenButler 2025.03	91.46	100	100	-	100	96.88	-	94.27	98.96	-	77.08	83.33	57.29	89.93	-	-	-	-	-	-	-
Qwen3-4B-SFT 2026.02	90.92	-	-	-	-	-	-	-	-	-	-	-	-	59.11	66.2	20.2	-	-	-	-	-
SnapKV 2026.05	90	-	-	-	-	-	-	-	-	60.08	72	-	-	75.81	98.73	-	47.7	68.15	91.15	60.5	94
TailorKV-2 2025.05	88	100	100	100	100	96	68	97	98	19.6	62.7	60	56	80.4	-	-	-	-	-	-	-
xKV 2025.03	86.67	100	96.88	-	97.92	97.92	-	96.62	96.09	-	78.47	78.13	56.25	88.5	-	-	-	-	-	-	-
Qwen3-8B-SFT w/ LongR 2026.02	86.64	-	-	-	-	-	-	-	-	-	-	-	-	71.98	69.88	59.41	-	-	-	-	-
TailorKV-1 2025.05	85.6	100	100	96	100	96	28	99	100	18.4	57.3	56	48	75.7	-	-	-	-	-	-	-
Minference 2026.05	83.48	-	-	-	-	-	-	-	-	61.84	82.73	-	-	65.61	86.53	-	42.8	29.67	82.25	81.1	40.07
Quest 2025.05	80	88	100	60	92	72	0	93	90	8.4	70.6	52	52	66	-	-	-	-	-	-	-
Quest 2025.03	77.71	90.75	90.63	-	96.88	87.5	-	85.42	94.27	-	81.94	83.33	57.29	84.57	-	-	-	-	-	-	-
SnapKV 2025.05	74.4	96	84	0	88	32	0	40	69	0.8	41.3	56	48	48.4	-	-	-	-	-	-	-
KVzip 2025.03	73.96	100	100	-	8.33	70.83	-	78.12	52.6	-	75.35	62.5	58.33	68	-	-	-	-	-	-	-
RTPurbo 2026.05	69.6	-	-	-	-	-	-	-	-	59.92	62	-	-	70.53	76.53	-	56	64.27	98.15	97.6	50.66
Qwen3-8B-SFT w/ DAPO 2026.02	58.52	-	-	-	-	-	-	-	-	-	-	-	-	55.02	66.13	40.4	-	-	-	-	-
Qwen3-4B-Base 2026.02	56.4	-	-	-	-	-	-	-	-	-	-	-	-	45.42	60.55	19.3	-	-	-	-	-
PQCache 2025.05	52.8	36	60	12	68	48	4	16	37	0	73.3	56	48	39.2	-	-	-	-	-	-	-
TailorKV-2 2025.05	42.4	100	100	92	100	84	28	62	90	33.6	48	28	48	65.8	-	-	-	-	-	-	-
TailorKV-2 2025.05	42.4	100	100	100	100	24	28	40	67	0.8	57.3	32	24	55.1	-	-	-	-	-	-	-
PQCache 2025.05	31.2	32	56	4	36	16	0	7	39	6	66.6	20	36	26.9	-	-	-	-	-	-	-
Qwen3-8B-SFT 2026.02	31	-	-	-	-	-	-	-	-	-	-	-	-	44.25	62.96	38.8	-	-	-	-	-
TailorKV-1 2025.05	28.8	100	100	100	96	12	24	41	65	1.2	58.7	28	24	52.2	-	-	-	-	-	-	-
Qwen3-8B-Base 2026.02	28.08	-	-	-	-	-	-	-	-	-	-	-	-	43.18	61.76	39.7	-	-	-	-	-
Yi-6B 2025.05	24.8	100	100	100	96	56	24	39	76	0.8	73.3	32	24	57.3	-	-	-	-	-	-	-
SnapKV 2025.05	22.4	80	28	0	20	4	0	11	11	5.6	48	24	44	22.9	-	-	-	-	-	-	-
Quest 2025.05	20	72	84	0	52	20	0	28	30	1.6	56	24	20	31.3	-	-	-	-	-	-	-
Quest 2025.05	16	68	92	20	68	40	0	24	42	10.8	62.6	28	36	39	-	-	-	-	-	-	-
SnapKV 2025.05	15.2	88	4	0	16	0	0	5	7	0	65.3	28	20	19.1	-	-	-	-	-	-	-
Yi-9B 2025.05	12.8	100	100	100	100	92	48	61	88	15.6	88	32	48	68.1	-	-	-	-	-	-	-
PQCache 2025.05	10.4	16	20	0	28	8	0	5	3	0	50.6	24	24	14.5	-	-	-	-	-	-	-
TailorKV-1 2025.05	6.4	100	100	92	100	84	28	53	89	32	49.3	32	48	62.6	-	-	-	-	-	-	-
StreamLLM 2025.05	2.4	8	4	0	8	0	0	5	3	0.8	72	28	28	12.2	-	-	-	-	-	-	-
StreamLLM 2025.05	0	0	4	0	4	0	0	1	0	1.2	74.6	16	28	9.9	-	-	-	-	-	-	-
StreamLLM 2025.05	0	0	0	0	8	0	0	3	0	0.4	62.6	20	16	8.5	-	-	-	-	-	-	-