Our new X account is live! Follow @wizwand_team for updates

Long-sequence modeling on Long Range Arena (LRA) v1 (test)

63.14ListOps

Mega

Updated 4d ago

Evaluation Results

Method
Mega 2022.08	63.14	90.43	91.25	90.44	96.01	97.98	88.21
Mega 2022.08	63.14	90.43	91.25	90.44	96.01	97.98	88.21
Liquid-S4 2022.08	62.75	89.02	91.2	89.5	94.8	96.66	87.32
Liquid-S4 2022.08	62.75	89.02	91.2	89.5	94.8	96.66	87.32
S5 2022.08	62.15	89.31	91.4	88	95.33	98.58	87.46
S5 2022.08	62.15	89.31	91.4	88	95.33	98.58	87.46
DSS-Softmax 2022.08	60.6	84.8	87.8	85.7	84.6	87.8	81.88
S4D-Lin 2022.06	60.52	86.97	90.96	87.93	93.96	-	78.39
S4D-Lin 2022.08	60.52	86.97	90.96	87.93	93.96	-	78.39
S4D-LegS 2022.06	60.47	86.18	89.46	88.19	93.06	91.95	84.89
S4D-LegS 2022.08	60.47	86.18	89.46	88.19	93.06	91.95	84.89
S4D-LegS 2022.08	60.47	86.18	89.46	88.19	93.06	91.95	84.89
S4-LegS/FouT 2022.08	60.45	86.78	90.3	89	94.44	-	78.5
Converter 2025.02	60.38	86.44	83.41	61.02	88.43	-	75.94
S4D-Inv 2022.06	60.18	87.34	91.09	87.83	93.78	92.8	85.5
S4D-Inv 2022.08	60.18	87.34	91.09	87.83	93.78	92.8	85.5
S5-Inv 2022.08	60.07	87.77	91.26	86.41	93.42	97.54	86.08
S5-Lin 2022.08	59.98	88.15	91.31	86.05	94.31	65.6	80.9
S4-LegS 2022.06	59.6	86.82	90.9	88.65	94.2	96.35	86.09
S4-LegS 2022.08	59.6	86.82	90.9	88.65	94.2	96.35	86.09
S4-LegS 2022.08	59.6	86.82	90.9	88.65	94.2	96.35	86.09
Mega-chunk 2022.08	58.76	90.19	90.97	85.8	94.41	93.81	85.66
Mega-chunk 2022.08	58.76	90.19	90.97	85.8	94.41	93.81	85.66
S4 (original) 2022.06	58.35	76.02	87.09	87.26	86.05	88.1	80.48
S4-FouT 2022.06	57.88	86.34	89.66	89.07	94.46	-	77.9
S4-FouT 2022.08	57.88	86.34	89.66	89.07	94.46	-	77.9
H-Transformer-1D 2022.08	49.53	78.69	63.99	46.05	68.78	-	61.41
H-Trans.-1D 2022.08	49.53	78.69	63.99	46.05	68.78	-	61.41
CCNN 2022.08	43.6	84.08	-	88.9	91.51	-	68.02
CCNN 2022.08	43.6	84.08	-	88.9	91.51	-	68.02
Paramixer 2025.02	39.71	78.87	78.73	44.68	79.16	-	58.91
Luna-256 2025.02	37.98	65.78	79.56	47.86	78.55	-	61.95
cosFormer 2025.02	37.9	63.41	61.36	43.17	70.33	-	55.23
Reformer 2022.05	37.27	56.1	53.4	38.07	68.5	-	50.67
Reformer 2022.08	37.27	56.1	53.4	38.07	68.5	-	50.56
Reformer 2025.02	37.27	56.1	53.4	38.07	68.5	-	50.67
Luna 2022.08	37.25	64.57	79.29	47.38	77.72	-	59.37
Luna-256 2022.08	37.25	64.57	79.29	47.38	77.72	-	59.37
Nyströmformer 2022.08	37.15	65.52	79.56	41.58	70.94	-	57.46
Nyströmformer 2025.02	37.15	65.52	79.56	41.58	70.94	-	58.95
Transformer + TLB 2022.05	37.05	81.88	76.91	57.51	79.06	-	66.48
Synthesizer 2022.05	36.99	61.68	54.67	41.61	69.45	-	52.88
Synthesizer 2025.02	36.99	61.68	54.67	41.61	69.45	-	52.88
Transformer 2022.06	36.37	64.27	57.46	42.44	71.4	-	53.66
Transformer 2022.08	36.37	64.27	57.46	42.44	71.4	-	53.66
Transformer 2022.08	36.37	64.27	57.46	42.44	71.4	-	53.66
Vanilla Trans. 2025.02	36.37	64.27	57.46	42.44	71.4	-	54.39
BigBird 2022.05	36.05	64.02	59.29	40.83	74.87	-	55.01
BigBird 2022.08	36.05	64.02	59.29	40.83	74.87	-	54.17
BigBird 2025.02	36.05	64.02	59.29	40.83	74.87	-	55.01
Linformer 2022.05	35.7	53.94	52.27	38.56	76.34	-	51.36
Linformer 2025.02	35.7	53.94	52.27	38.56	76.34	-	51.36
Longformer 2022.05	35.63	62.85	56.89	42.22	69.71	-	53.46
Longformer 2025.02	35.63	62.85	56.89	42.22	69.71	-	53.46
FNet 2022.08	35.33	65.11	59.61	38.67	77.8	-	54.42
FNet 2025.02	35.33	65.11	59.61	38.67	77.8	-	55.3
Sinkhorn Transformer 2022.05	33.67	61.2	53.83	41.23	67.45	-	51.29
Sinkhorn Trans. 2025.02	33.67	61.2	53.83	41.23	67.45	-	51.29
Performer 2022.05	18.01	65.4	53.82	42.77	77.05	-	51.41
Performer 2022.08	18.01	65.4	53.82	42.77	77.05	-	51.18
Performer 2025.02	18.01	65.4	53.82	42.77	77.05	-	51.41
Sparse Transformer 2022.05	17.07	63.58	59.59	44.24	71.71	-	51.24
Sparse Trans. 2025.02	17.07	63.58	59.59	44.24	71.71	-	51.24
Linear Transformer 2022.05	16.13	65.9	53.09	42.34	75.3	-	50.55
Linear Trans. 2022.08	16.13	65.9	53.09	42.34	75.3	-	50.46
Linear Trans. 2025.02	16.13	65.9	53.09	42.34	75.3	-	50.55