Share your thoughts, 1 month free Claude Pro on usSee more

Language Modeling on LM-Harness (Standard Suite Metrics)

32.85ARC-C Accuracy

DeltaNet

Updated 2mo ago

Evaluation Results

Method	Links
DeltaNet 2025.11		32.85	58.16	42.51	81	61.13	73.78	43.9	61.72	11.8	46.08	51.29
Gated DeltaNet 2025.11		32.59	60.02	62.75	82	62.8	74.32	80.6	62.35	8.26	44.28	57
Gated KalmaNet 2025.11		32.51	59.89	61.68	85	63.84	74.81	83.2	64.17	12.89	50.95	58.89
Transformer 2025.11		32.25	56.1	64.28	80	60.96	73.56	79.5	61.72	58.53	72.28	63.92
Mamba2 2025.11		32.24	59.64	58.72	82	62.23	73.78	79.8	62.19	7.71	41.13	55.94
Mamba2 2025.11		28.07	46.63	60.21	70	44.57	67.57	65.5	54.3	1.45	15.75	45.4
Gated Linear Attention 2025.11		27.82	50.8	52.57	78	48.83	70.13	69.6	54.54	2.81	20.43	47.55
DeltaNet 2025.11		27.56	46.25	59.97	71	43.18	67.74	65.9	55.41	3.09	20.61	46.07
Gated DeltaNet 2025.11		27.05	47.98	59.54	74	44.27	67.36	66.2	53.83	2.18	17.82	46.02
Transformer 2025.11		26.62	46.42	59.94	77	44.01	67.14	68.3	54.06	8.35	45.18	49.7
Gated Linear Attention 2025.11		25.94	42	58.84	70	36.34	63.6	58.2	51.85	1.45	10.53	41.88
Gated KalmaNet 2025.11		25.43	46.55	60.73	74	44.59	68.88	67.6	52.41	6.17	21.87	46.82
Gated DeltaNet 2025.11		25.17	41.96	58.23	72	36.96	64.69	63.6	51.7	1.91	11.88	42.81
DeltaNet 2025.11		25.09	41.92	61.13	65	37.2	64.47	64	49.49	2.81	14.31	42.54
Gated KalmaNet 2025.11		24.57	43.22	56.94	71	37.22	64.47	62.8	50.83	1.45	14.04	42.65
Transformer 2025.11		24.4	42.26	59.88	70	36.19	64.15	61.5	51.7	5.17	35.64	45.09
Gated Linear Attention 2025.11		24.06	40.28	56.57	71	32.7	62.24	57.8	50.67	1	9.18	40.55