Our new X account is live! Follow @wizwand_team for updates

Linguistic Minimal Pair Scoring on BLiMP

88.9Overall Accuracy

Head Ensemble

Updated 4d ago

Evaluation Results

Method
Head Ensemble 2022.05	88.9	97	83.3	86.9	81.6	93.7	88.1	88.8	95.5	83	94.2	84.2	92.7	-
Human Baseline 2022.05	88.6	97.5	90	87.3	83.9	92.2	85	86.9	97	84.9	88.1	86.6	90.9	-
Head Ensemble 2022.05	87.8	96.3	79.6	87.6	82.6	93.6	84.9	90.4	94.3	83	94.6	80.6	92.8	-
RoBERTa (large) 2023.05	87.5	-	-	-	-	-	-	-	-	-	-	-	-	-
Phenomenon Head 2022.05	86.8	97.6	80.7	90.3	80.7	91.9	90.4	86.8	95.9	78.9	91.3	82.9	90.4	-
RoBERTa (base) 2023.05	86.7	-	-	-	-	-	-	-	-	-	-	-	-	-
ROBERTa-large 2022.05	86.5	97.8	84.6	79.1	84.1	96.8	90.8	88.9	96.8	83.4	85.5	70.2	91.4	-
Phenomenon Head 2022.05	86.5	97.6	79.9	90.5	80.7	91.6	89.9	87.1	95.9	78.9	91.1	83.4	90.2	-
RoBERTa (large) 2022.05	86.5	97.8	84.6	79.1	84.1	96.8	90.8	-	96.8	83.4	85.5	70.2	91.4	88.9
RoBERTa (large) 2023.05	86.5	-	-	-	-	-	-	-	-	-	-	-	-	-
RoBERTa (large) 2023.05	85.9	-	-	-	-	-	-	-	-	-	-	-	-	-
Head Ensemble 2022.05	85.8	93.9	82.5	85.6	77	96.3	88.1	80.7	95.7	77	92.5	83.8	88.9	-
ROBERTa-base 2022.05	85.4	97.3	83.5	77.8	81.9	97	91.4	90.1	96.2	80.7	81	69.8	91.9	-
RoBERTa (base) 2022.05	85.4	97.3	83.5	77.8	81.9	97	91.4	-	96.2	80.7	81	69.8	91.9	90.1
RoBERTa (base) 2023.05	85.4	-	-	-	-	-	-	-	-	-	-	-	-	-
RoBERTa (base) 2023.05	85.4	-	-	-	-	-	-	-	-	-	-	-	-	-
Transcormer (base) 2022.05	85	98.1	80	84.8	79.2	96.2	89.3	-	96.1	76.6	87.8	75	91.7	84.6
BERT (large) 2023.05	85	-	-	-	-	-	-	-	-	-	-	-	-	-
BERT-large 2022.05	84.8	97.2	80.7	82	82.7	97.6	86.4	84.3	92.8	77	83.4	72.8	91.9	-
BERT (large) 2022.05	84.8	97.2	80.7	82	82.7	97.6	86.4	-	92.8	77	83.4	72.8	91.9	84.3
BERT (large) 2023.05	84.8	-	-	-	-	-	-	-	-	-	-	-	-	-
BERT (base) 2023.05	84.7	-	-	-	-	-	-	-	-	-	-	-	-	-
Transcormer (base) 2022.05	84.6	98.1	80.7	83.2	80.2	96	90.7	-	95.5	74.3	85.7	73.4	91.3	84.1
Head Ensemble 2022.05	84.3	93.3	79.9	83.5	78.6	96.4	78.4	79.5	93.8	74.4	92.5	81.7	86.8	-
BERT-base 2022.05	84.2	97	80	82.3	79.6	97.6	89.4	83.1	96.5	73.6	84.7	71.2	92.4	-
BERT (base) 2022.05	84.2	97	80	82.3	79.6	97.6	89.4	-	96.5	73.6	84.7	71.2	92.4	83.1
BERT (base) 2023.05	84.2	-	-	-	-	-	-	-	-	-	-	-	-	-
BERT (base) 2023.05	83.1	-	-	-	-	-	-	-	-	-	-	-	-	-
GPT2-medium 2022.05	82.6	99.4	83.4	77.8	83	96.3	86.3	81.3	94.9	71.7	74.7	74.1	88.3	-
GPT-2 2022.05	82.6	99.4	83.4	77.8	83	96.3	86.3	-	94.9	71.7	74.7	74.1	88.3	81.3
BERT (large) 2023.05	82.6	-	-	-	-	-	-	-	-	-	-	-	-	-
Top Head [11; 10] 2022.05	81.9	90.1	66	84.7	71.7	91	86.7	87.1	89.5	76.8	90.7	78.4	85	-
Phenomenon Head 2022.05	81.8	94.5	75.8	80.4	79.2	96.7	89.1	75	93	72.2	84.4	81.2	82	-
Phenomenon Head 2022.05	81.7	94.9	75.9	80.4	79.2	96.7	89.1	75.9	93	70.5	84.6	81.2	82.1	-
GPT2-large 2022.05	81.5	99.6	78.3	80.1	80.5	93.3	86.6	81.3	84.1	70.6	78.9	71.3	89	-
Top Head [11; 10] 2022.05	80.8	88.8	63.3	83.7	69.8	90.2	88.8	86.8	89.1	76.5	89	78.4	83.1	-
UNION 2026.01	75.5	-	-	-	-	-	-	-	-	-	-	-	-	-
Top Head [8; 0] 2022.05	75.4	86.8	75.9	63.4	79.2	83.7	72.2	67.3	90.3	70	83.1	63.5	82.1	-
Top Head [8; 0] 2022.05	75.4	86.8	75.8	63.3	79.2	83.6	72.1	67.3	90.2	70.2	83.1	63.6	82	-
All Heads 2022.05	74.3	80.6	71.2	78.6	67.8	90.6	89.9	75.5	65.1	73.4	65.1	57	75.6	-
All Heads 2022.05	74.3	81.1	66.6	76.8	65.2	89.9	91.1	74.6	63.3	74.1	63.2	56.9	74.9	-
No Procedural Pretraining 2026.01	73.2	-	-	-	-	-	-	-	-	-	-	-	-	-
UNION 2026.01	72.9	-	-	-	-	-	-	-	-	-	-	-	-	-
No Procedural Pretraining 2026.01	71.5	-	-	-	-	-	-	-	-	-	-	-	-	-
LSTM 2022.05	69.8	91.7	73.2	73.5	67	85.4	67.6	73.9	89.1	46.6	51.7	64.5	80.1	-
Transformer-XL 2022.05	69.6	94.1	72.2	74.7	71.5	83	77.2	66.6	78.2	48.4	55.2	69.3	76	-
All Heads 2022.05	65.3	77.8	68.5	63.2	63.6	86	73.4	48.4	91	51.3	62	51.2	72.6	-
All Heads 2022.05	64.8	79.6	69.1	63.9	62.6	86.2	70.7	47.3	90.7	49.5	61.1	50	72	-
5-gram 2022.05	61.2	47.9	71.9	64.4	68.5	70	36.9	60.2	79.5	57.2	45.5	53.5	60.3	-