Share your thoughts, 1 month free Claude Pro on usSee more

Long-context language modeling evaluation on RULER

100Single-key Accuracy

Full Attention

Updated 5d ago

Evaluation Results

Method
Full Attention 2026.03	100	100	100	100	82.5	84.4	89.8	59.2	89.5	-	-
LycheeCluster 2026.03	100	100	100	99.5	81.2	80.3	89.8	59.2	88.8	-	-
LycheeCluster 2026.03	100	100	96.4	100	78	83.7	81.6	59.2	87.4	-	-
Full Cache 2025.02	100	100	99.75	98.25	99.4	95.33	84	62	91.05	44.9	0
SnapKV 2025.02	100	93	87.5	92.75	96.6	90	41	52	67.93	20.3	25.4
SnapKV + CriticalKV 2025.02	100	100	99	99	97	92.67	63	53	76.89	33.9	15.6
AdaKV + CriticalKV 2025.02	100	100	98.75	99.25	99.6	94.67	64	58	86.28	52.4	5.2
HeadKV 2025.02	100	99	93	97	98	92.67	56	54	79.98	17.1	12.2
HeadKV + CriticalKV 2025.02	100	100	99	99.5	99.4	93.33	72	59	89.29	56.5	1.9
Full Cache 2025.02	100	99	99.25	100	100	96	82	74	93.91	90.6	0
SnapKV 2025.02	100	63	61.75	71.75	99.8	92	50	64	63.86	87.9	32
SnapKV + CriticalKV 2025.02	100	98	99.5	99.75	100	93.33	67	69	81.09	88.6	13.7
AdaKV 2025.02	100	81	83.5	87.5	100	93.33	55	64	71.09	88.9	24.3
AdaKV + CriticalKV 2025.02	100	98	99.5	100	100	94.33	64	66	83.87	89.5	10.7
HeadKV 2025.02	100	92	96.25	97.25	100	94.67	68	66	81.04	89.3	13.7
HeadKV + CriticalKV 2025.02	100	99	98.75	100	100	95.33	72	69	90.69	89.9	3.4
AdaKV 2025.02	99	98	98.25	97.5	98.6	92	43	54	78.38	26.6	13.9
Full Cache 2025.02	99	94	95	94.25	90.6	95.67	60	59	78.27	30	0
AdaKV + CriticalKV 2025.02	99	89	95	85	89.4	95.33	60	57	69.17	40.5	11.6
Full Attention 2026.03	95.9	95.9	97.5	97.5	76.3	76.9	79.6	59.2	84.8	-	-
LycheeCluster 2026.03	95.9	98	92.9	93.9	85.7	76.2	75.5	59.2	84.7	-	-
LycheeCluster 2026.03	93.9	100	94.9	94.9	73.1	89.8	77.6	51	84.4	-	-
Full Attention 2026.03	91.8	100	100	99	79.6	87.1	81.6	59.2	87.3	-	-
HeadKV + CriticalKV 2025.02	84	53	62.25	60.5	81.4	95	53	55	57.59	49.5	26.4
SnapKV + CriticalKV 2025.02	74	31	24.25	26	81.8	94.67	45	53	41.56	48.5	46.9
HeadKV 2025.02	70	24	16.5	16.25	82.6	92.67	45	53	39.59	30.6	49.4
SnapKV 2025.02	65	16	8	11.25	74.8	91.67	34	51	32.15	32.2	58.9
Full Attention 2026.03	63.3	100	98.5	98.5	82	93.9	79.6	51	83.3	-	-
AdaKV 2025.02	44	25	14.25	20	71.6	92.33	44	55	34.88	25.3	55.4