Our new X account is live! Follow @wizwand_team for updates

Natural Language Understanding on SuperGLUE (test)

92.4BoolQ Accuracy

ST-MoE-32B

Updated 2d ago

Evaluation Results

Method	Links
ST-MoE-32B 2022.12		92.4	91.2	98	96.9	99.2	93.5	77.7	96.6	-	-	94.4	95.1	89.6	65.8	-	-	-
ST-MoE-32B 2022.04		92.4	91.2	96.9	-	99.2	93.5	77.7	96.6	89.6	-	95.1	-	-	-	-	-	-
FLAN-1B 2024.03		92.3	-	92.5	-	97.8	92.1	69.9	70.7	88.3	-	38.9	-	-	-	-	84.8	21.9
FLAN-3B 2024.03		92.1	-	92.7	-	99.3	96	79.7	80.2	90.4	-	62.2	-	-	-	-	95.3	22
Turing NLR v5 2022.12		92	90.9	97.6	95.9	98.2	94.1	77.1	97.3	-	-	95.9	96.4	88.4	63	-	-	-
PaLM 540B 2022.12		91.9	90.4	96	94.4	99	94.1	77.4	95.9	-	-	93.3	94.2	88.7	63.6	-	-	-
PaLM 540B 2022.04		91.9	90.4	94.4	-	99	95.9	77.4	95.9	88.7	-	94.2	-	-	-	-	-	-
T5-11B 2019.10		91.2	88.9	96.8	93.9	94.8	92.5	76.9	93.8	-	88.1	93.4	94.1	-	63.3	-	-	-
Fine-tuned SOTA 2020.05		91	89	96.9	93.9	94.8	92.5	76.1	93.8	62.3	88.2	92.5	93.3	-	-	-	-	-
ERNIE 3.0 2022.12		91	90.6	99.2	98.6	97.4	92.6	77.4	97.3	-	-	94.2	94.7	88.6	63.2	-	-	-
Vega v2 2022.12		90.5	91.3	99.2	98.6	99.4	96	77.4	98.6	-	-	93.9	94.4	88.2	62.4	-	-	-
T5-3B 2019.10		89.9	86.4	94.4	90.3	92	90.7	72.1	90.4	-	86.8	90.4	91.2	-	58.3	-	-	-
SuperGLUE Human Baselines 2022.12		89	89.8	98.9	95.8	100	93.6	80	100	-	-	91.3	91.7	81.8	51.9	-	-	-
Previous best 2019.10		87.1	84.6	95.2	90.5	90.6	88.2	69.9	89	-	84.4	90	90.6	-	52.5	-	-	-
Adam 2024.04		86	-	-	-	59	87	70	68	76	-	-	-	-	-	-	-	-
T5-Large 2019.10		85.4	82.3	94.8	91.6	83.4	87.8	69.3	86.3	-	83.3	85.9	86.8	-	50.7	-	-	-
BAdam 2024.04		85	-	-	-	69	76	64	65	77	-	-	-	-	-	-	-	-
T5-Base 2019.10		81.4	76.2	94	86.2	71.2	81.5	68.3	80.8	-	79.7	74.2	75	-	43.1	-	-	-
LoRA 2024.04		81	-	-	-	56	79	59	62	69	-	-	-	-	-	-	-	-
ByT5 2023.07		79.2	-	91.07	90.37	58	81.23	70.85	-	-	-	-	72.79	70.14	32	72.1	-	-
SuperGLUE Baselines 2022.12		79	71.5	90.4	84.8	73.8	79	69.6	64.4	-	-	71.3	72	70	24.1	-	-	-
mT5 2023.07		78.1	-	92.86	90.53	61	74.37	68.03	-	-	-	-	68.26	73.03	33.68	67.22	-	-
Fine-tuned BERT-Large 2020.05		77.4	69	83.6	75.7	70.6	71.7	69.6	64.6	24.1	70	71.3	72	-	-	-	-	-
GPT-3 Few-Shot 2020.05		76.4	71.8	75.6	52	92	69	49.4	80.1	30.5	75.4	90.2	91.1	-	-	-	-	-
T5-Small 2019.10		76.4	63.3	81.6	56.9	46	73.3	66.9	70.5	-	69.3	55.4	56.3	-	26.3	-	-	-
Best Decoder-only LM 2022.04		76.4	71.8	52	-	92	69	49.4	80.1	75.4	-	91.1	-	-	-	-	-	-
HyperMoE 2024.02		73.14	64.88	69.68	-	45	67.67	65.31	56.53	67.68	-	74.06	-	-	-	-	-	-
MoE 2024.02		72.69	64.04	69.64	-	45	67.15	63.63	56.58	66.38	-	71.26	-	-	-	-	-	-
MoE-Share 2024.02		72.11	64.21	67.85	-	45	67.87	65.36	56.84	66.71	-	71.91	-	-	-	-	-	-
Tk-Instruct-3B 2024.03		68.8	-	66.3	-	73.8	62.7	50.4	51.2	59.9	-	18.6	-	-	-	-	53.5	49.9
Tk-CoAT-1B 2024.03		63.8	-	46.9	-	76.1	53.7	53.5	53.6	11.4	-	17	-	-	-	-	50.4	52.7
Tk-CoAT-3B 2024.03		63.6	-	60.4	-	81.3	52	56.9	53.6	56.9	-	23.1	-	-	-	-	57.9	57.2
Tk-Instruct-1B 2024.03		63.4	-	46	-	76.9	55.5	53.5	49.8	62.2	-	13.1	-	-	-	-	51.9	57.2
GPT-Neo-125M 2022.12		62.2	50.8	41.1	-	60	53.1	53.6	36.5	59.9	-	39.6	-	-	-	-	-	-
GPT-Neo-125M 2022.12		62.1	47.2	17.9	-	60	53.1	50	38.5	56.3	-	39.6	-	-	-	-	-	-
Hybrid H3-1.3B 2022.12		61.7	56.5	33.9	-	74	53.4	50.3	52.9	58.2	-	67.8	-	-	-	-	-	-
Hybrid H3-355M 2022.12		61.5	54.7	41.1	-	69	55.2	51.7	37.5	59.5	-	62.3	-	-	-	-	-	-
GPT-Neo-125M 2022.12		61.2	47	42.9	-	68	53.1	50	38.5	22.5	-	39.7	-	-	-	-	-	-
GPT-Neo-2.7B 2022.12		61.1	54.6	50	-	67	52.3	50	37.5	59.1	-	60	-	-	-	-	-	-
Hybrid H3-2.7B 2022.12		61.1	56.8	37.5	-	81	57	51.3	36.5	58.7	-	71.3	-	-	-	-	-	-
GPT-2 medium (355M) 2022.12		61	52.6	28.6	-	65	51.3	52	50	59.5	-	53.3	-	-	-	-	-	-
OPT-350M 2022.12		60.9	53.1	35.7	-	60	53.4	50.8	53.5	58.9	-	51.4	-	-	-	-	-	-
OPT-2.7B 2022.12		60.9	55.5	41.1	-	66	50.5	50.8	51	57.4	-	65.9	-	-	-	-	-	-
T0-3B 2024.03		60.1	-	48	-	56.8	51.3	54	53.5	56.2	-	20.5	-	-	-	-	65	36.1
GPT-Neo-1.3B 2022.12		59.9	52.1	33.9	-	66	52.3	50	41.3	57.9	-	55.5	-	-	-	-	-	-
OPT-125M 2022.12		59.6	50.1	37.4	-	60	48.7	52	39.4	58.9	-	44.9	-	-	-	-	-	-
Hybrid H3-125M 2022.12		59.6	53.9	48.2	-	67	59.2	51.4	39.4	51.4	-	55	-	-	-	-	-	-
GPT-Neo-125M 2022.12		56.4	33.6	8.9	-	60	45.8	11.3	27.9	19.1	-	39.6	-	-	-	-	-	-
Hybrid H3-125M 2022.12		56.1	53.7	51.8	-	67	58.1	49.1	43.3	48.9	-	55	-	-	-	-	-	-
OPT-1.3B 2022.12		55	52.9	39.3	-	69	53.4	49.5	36.5	58.3	-	61.8	-	-	-	-	-	-
Hybrid H3-125M 2022.12		49.5	50.5	41.1	-	67	58.1	49.1	43.3	40.3	-	55.2	-	-	-	-	-	-
Hybrid H3-125M 2022.12		47.6	28.8	8.9	-	67	47.3	0	0	4.4	-	55	-	-	-	-	-	-
OPT-125M 2022.12		45.9	41.7	8.9	-	60	49.8	48.4	36.5	39.1	-	44.9	-	-	-	-	-	-
Hybrid H3-355M 2022.12		45.4	51.9	37.5	-	73	50.5	47.5	42.3	57.5	-	61.4	-	-	-	-	-	-
Hybrid H3-355M 2022.12		44.4	36.6	8.9	-	69	47.3	21.5	22.1	17.1	-	62.3	-	-	-	-	-	-
OPT-125M 2022.12		41.9	47.9	44.6	-	60	47.3	50.2	36.5	57.9	-	44.9	-	-	-	-	-	-
OPT-350M 2022.12		39.7	39.6	16.1	-	60	49.5	34.8	41.3	23.6	-	51.4	-	-	-	-	-	-
OPT-350M 2022.12		39.4	45.6	41.1	-	59	46.2	50	37.5	40.6	-	51.3	-	-	-	-	-	-
OPT-125M 2022.12		38.5	43.2	33.9	-	60	47.3	49.1	36.5	35.5	-	44.8	-	-	-	-	-	-
GPT-2 medium 2022.12		38.4	38.1	21.4	-	65	16.2	50.2	50	10.5	-	53.3	-	-	-	-	-	-
H3-125M 2022.12		37.8	43.9	48.2	-	51	52.3	50	63.5	32.6	-	15.8	-	-	-	-	-	-
H3-125M 2022.12		37.8	20.3	8.9	-	53	47.3	0	0	0	-	15.4	-	-	-	-	-	-
GPT-2 medium (355M) 2022.12		37.8	44	28.6	-	63	47.3	50.5	36.5	35.3	-	53.1	-	-	-	-	-	-