Our new X account is live! Follow @wizwand_team for updates

Language Modeling on One Billion Word Benchmark (test)

20.25Test Perplexity

H-Transformer-1D

Updated 3d ago

Evaluation Results

Method
H-Transformer-1D 2021.07	20.25	-
Transformer-XL Large 2019.01	21.8	-
Transformer-XL 2021.07	21.8	-
Adaptive inputs (very large) 2018.09	23.02	-
Baevski and Auli 2021.07	23.02	-
BIG GLSTM-G4 2016.12	23.3	-
Transformer-XL Base 2019.01	23.5	-
Transformer-XL 2021.07	23.5	-
10 LSTMs + SNM10-SKIP 2016.02	23.7	-
Józefowicz et al. (2016) 2018.08	23.7	-
10 LSTMs + SNM10-SKIP 2018.09	23.7	-
Adaptive Input 2019.01	23.7	-
Adaptive inputs (large) 2018.09	23.91	-
Baevski and Auli 2021.07	23.91	-
H-Transformer-1D 2021.07	23.95	-
Mesh Tensorflow 2019.01	24	-
Shazeer et al. 2021.07	24	-
Adaptive Input 2019.01	24.1	-
Transformer baseline 2021.07	24.8	-
10 LSTMs + KN-5 2016.02	25.1	-
Adaptive inputs 2018.09	25.22	-
10 LSTMs + KN-5 2016.02	25.3	-
Char-CNN 2018.09	25.88	-
10 Best LSTM Models 2016.02	26.1	-
10 Best LSTM Models 2016.02	26.3	-
Shazeer et al. (2017) 2018.09	28	-
High-Budget MoE 2019.01	28	-
BIG LSTM+CNN INPUTS 2016.02	30	-
CNN-8192-1024 2018.04	30	-
Józefowicz et al. (2016) 2018.09	30	-
LSTM + CNN Input 2019.01	30	-
Transformer baseline 2021.07	30.04	-
2-LAYER LSTM-8192-1024 (BIG LSTM) 2016.02	30.6	-
2-layer LSTM-8192-1024 2016.12	30.6	-
8192-1024 2018.04	30.6	-
LSTM 2019.01	30.6	-
GCNN-14 Bottleneck 2016.12	31.9	-
Dauphin et al. (2017) 2018.09	31.9	-
GCNN-14 bottleneck 2019.01	31.9	-
Transformer 2023.10	31.98	-
LSTM-8192-2048 (50% DROPOUT) 2016.02	32.2	-
SEDD Absorb 2023.10	32.79	-
Low-Budget MoE 2019.01	34.1	-
BIG LSTM+CNN INPUTS + CNN SOFTMAX + 128-DIM CORRECTION 2016.02	35.8	-
G-LSTM-2 2019.01	36	-
MoS 2017.11	37.1	-
LSTM-8192-2048 (NO DROPOUT) 2016.02	37.9	-
GCNN-13 2016.12	38.1	-
CNN-4096-512 2018.04	39.7	-
BIG LSTM+CNN INPUTS + CNN SOFTMAX 2016.02	39.8	-
2-layer LSTM-2048 2016.12	39.8	-
2048-Adaptive 2018.04	39.8	-
SEDD Uniform 2023.10	40.25	-
vanilla LSTM 2018.04	40.27	-
T64 2018.08	40.6	1.03
Large Ensemble 2016.02	41	-
RNN+SNM10-SKIP 2016.02	41.3	-
RNN+KN-5 2016.02	42	-
RNN+KN-5 2016.02	42.4	-
Softmax 2017.11	42.77	-
LSTM-2048-512 2016.02	43.7	-
LSTM-2048-512 2016.12	43.7	-
Large Ensemble 2016.02	43.8	-
LSTM-2048 2016.12	43.9	-
LD-Net 2018.04	45.14	-
2048-512 2018.04	47.5	-
BIG LSTM+CNN INPUTS + CHAR LSTM PREDICTIONS 2016.02	47.9	-
LSTM-1024-512 2016.02	48.2	-
vanilla LSTM 2018.04	48.85	-
LD-Net 2018.04	50.06	-
RNN-1024 + MAXENT 9-GRAM FEATURES 2016.02	51.3	-
RNN-1024 + MaxEnt 9 Gram Features 2016.12	51.3	-
RNN-1024 + 9 Gram 2019.01	51.3	-
SPARSE NON-NEGATIVE MATRIX LM 2016.02	52.9	-
Sparse Non-Negative Matrix LM 2016.12	52.9	-
Sparse Non-Negative 2019.01	52.9	-
LSTM-512-512 2016.02	54.1	-
GPT-2 2023.10	55.72	-
KN+SNM 2014.12	61.4	-
KN+SNM 2014.12	61.7	-
KN 2014.12	62.9	-
KN+SNM 2014.12	63	-
KN 2014.12	63.2	-
DiffusionBert 2023.10	63.78	-
KN 2014.12	64.3	-
SNM 2014.12	64.8	-
SNM 2014.12	65.4	-
KN+SNM 2014.12	66.5	-
SNM 2014.12	67	-
KN 2014.12	67.6	-
INTERPOLATED KN 5-GRAM, 1.1B N-GRAMS 2016.02	67.6	-
Interpolated KN 5-Gram 2016.12	67.6	-
SIGMOID-RNN-2048 2016.02	68.3	-
Sigmoid-RNN-2048 2016.12	68.3	-
LSTM 2015.04	68.8	-
IRNN 2015.04	69.4	-
IRNN 2015.04	70.2	-
SNM 2014.12	70.8	-
RNN 2015.04	71.8	-
RNN 2015.04	72.5	-

Showing 100 of 108 rows