Our new X account is live! Follow @wizwand_team for updates

Language Modeling on WikiText-103 (test)

2.22Perplexity

RETRO

Updated 2d ago

Evaluation Results

Method
RETRO 2021.12	2.22	-	-	-	-	-
RETRO 2021.12	3.32	-	-	-	-	-
ADEPT 2026.01	5.5	-	-	-	54.9	-
ADEPT 2026.01	6.2	-	-	-	17.8	-
ADEPT 2026.01	6.6	-	-	-	20.5	-
MIDUS-HML 2025.12	7.4	-	-	-	-	-
MIDUS-HML 2025.12	7.5	-	-	-	-	-
LESA 2025.12	7.72	-	-	-	-	-
LESA 2025.12	7.73	-	-	-	-	-
OpT-DeUS 2025.12	7.73	-	-	-	-	-
OpT-DeUS 2025.12	7.73	-	-	-	-	-
Llama Pro 2025.12	7.81	-	-	-	-	-
Llama Pro 2025.12	7.81	-	-	-	-	-
Avg-DeUS 2025.12	7.91	-	-	-	-	-
Avg-DeUS 2025.12	7.95	-	-	-	-	-
ADEPT 2026.01	8.1	-	-	-	21.8	-
Base 2025.12	8.32	-	-	-	-	-
Base 2025.12	8.35	-	-	-	-	-
SOLAR 2025.12	9.68	-	-	-	-	-
Transformer 2023.12	9.8	-	-	-	-	-
SwitchAll 2023.12	9.81	-	-	-	-	-
SOLAR 2025.12	9.9	-	-	-	-	-
Baseline 2021.12	10.65	-	-	-	-	-
Megatron-LM 2019.09	10.81	-	-	-	-	-
SwitchAll 2023.12	12.17	-	-	-	-	-
Transformer 2023.12	12.32	-	-	-	-	-
Megatron-LM 2019.09	12.76	-	-	-	-	-
Ensemble of All 2023.11	13.29	-	-	-	-	-
GateLoop 2023.11	13.4	-	-	-	-	-
full-precision 2022.03	14.19	-	-	-	-	-
full-prec. 2022.03	14.2	-	-	-	-	-
kNN LM 2023.11	14.54	-	-	-	-	-
QuantGPT 2022.03	14.58	-	-	-	-	-
QuantGPT 2022.03	14.6	-	-	-	-	-
ADEPT 2026.01	14.7	-	-	-	21.4	-
LSQ 2022.03	15.24	-	-	-	-	-
QuantGPT 2022.03	15.3	-	-	-	-	-
QuantGPT 2022.03	15.31	-	-	-	-	-
adaptive kNN-LM 2022.10	15.5	-	-	-	-	-
Adaptive kNN-LM 2022.10	15.5	-	-	-	-	-
Adaptive kNN-LM (TFIDF) 2022.10	15.54	-	-	-	-	-
kNN-LM + Continuous Cache 2019.11	15.79	-	-	-	-	-
kNN-LM 2019.09	15.79	-	-	-	-	-
kNN-LM 2020.12	15.79	-	-	-	-	-
kNN-LM + CCache 2022.10	15.79	-	-	-	-	-
Routing Transformer 2020.03	15.8	-	-	-	-	-
Best Existing 2020.02	15.8	-	-	-	-	-
Routing Transformer 2020.12	15.8	-	-	-	-	-
Routing Trans. 2021.05	15.8	-	-	-	-	-
Routing Transformer 2022.02	15.8	-	-	-	-	-
Routing Transformer 2021.02	15.8	-	-	-	-	-
kNN-LM 2021.02	15.8	-	-	-	-	-
LAQ 2022.03	15.88	-	-	-	-	-
GPT2 + GFSA 2023.12	15.919	-	-	-	-	-
GPT2 2023.12	15.939	-	-	-	-	-
kNN-LM 2019.11	16.12	-	-	-	-	-
kNN-LM 2022.10	16.12	-	-	-	-	-
kNN-LM 2022.10	16.12	-	-	-	-	-
ADEPT 2026.01	16.3	-	-	-	29.6	-
LightPAFF 2022.03	16.4	-	-	-	-	-
PACT 2022.03	16.76	-	-	-	-	-
LAQ 2022.03	16.91	-	-	-	-	-
QuantGPT 2022.03	16.98	-	-	-	-	-
Segatron-XL large + HCP 2022.03	17	-	-	-	-	-
QuantGPT 2022.03	17	-	-	-	-	-
Compressive Transformer 2019.11	17.1	-	-	-	-	-
Compressive Transformer 2020.04	17.1	-	-	-	-	-
Segatron-XL large 2020.04	17.1	-	-	-	-	-
Compressive Transformer 2020.12	17.1	-	-	-	-	-
Compressive Trans. 2021.05	17.1	-	-	-	-	-
Compressive Transformer 2022.03	17.1	-	-	-	-	-
Segatron-XL large 2022.03	17.1	-	-	-	-	-
Compressive Transformer 2022.02	17.1	-	-	-	-	-
Transformer-XL 18L + Compressive memory 2021.02	17.1	-	-	-	-	-
SRU++ Large (d = 8 d') 2021.02	17.1	-	-	-	-	15
MEGALODON 2024.04	17.23	-	-	-	-	-
SRU++ Large (k = 5) 2021.02	17.3	-	-	-	-	11
Transformer-XL Large + Phrase Induction 2019.06	17.4	-	-	-	-	-
Phrase Induction 2019.11	17.4	-	-	-	-	-
SRU++ Large 2021.02	17.4	-	-	-	-	14
GPT-2 2019.04	17.48	-	-	-	-	-
Staged Training 2020.12	17.56	-	-	-	-	-
Adaptive Inputs + LayerDrop 2019.09	17.7	-	-	-	-	-
Trans.+LayerDrop 2021.05	17.7	-	-	-	-	-
Tensor-Transformer_1core + PN 2020.03	17.9	-	-	-	-	-
Tensor-Transformer 2020.03	17.9	-	-	-	-	-
Sandwich Transformer 2020.12	17.96	-	-	-	-	-
adaptive kNN-LM 2022.10	18.03	-	-	-	-	-
kNN-LM 2022.10	18.05	-	-	-	-	-
MEGA 2022.09	18.07	-	-	-	-	-
MEGA 2022.09	18.07	-	-	-	-	-
MEGA 2024.04	18.07	-	-	-	-	-
MEGA 2023.11	18.09	-	-	-	-	-
TransformerXL 2019.11	18.1	-	-	-	-	-
Shortformer 2020.12	18.15	-	-	-	-	-
Shortformer 2022.02	18.15	-	-	-	-	-
ShortFormer 2023.11	18.15	-	-	-	-	-
Transformer-XL 2022.02	18.2	-	-	-	-	-
Feedback Transformer 2021.02	18.2	-	-	-	-	214
Perceiver AR 2022.02	18.25	-	-	-	-	-

Showing 100 of 532 rows