Our new X account is live! Follow @wizwand_team for updates

Language Modeling on WikiText-103 (val)

1.01PPL

BiLSTM

Updated 2d ago

Evaluation Results

Method
BiLSTM 2025.12	1.01	-	-	-	0.01	0.02	99
Ensemble of All 2023.11	13.11	-	-	-	-	-	-
kNN LM 2023.11	14.39	-	-	-	-	-	-
Transformer-XL + RMS dynamic eval 2019.04	15.8	-	-	-	-	-	-
Compressive Transformer 2019.11	16	-	-	-	-	-	-
Compressive Transformer 2022.03	16	-	-	-	-	-	-
Compressive Transformer 2022.02	16	-	-	-	-	-	-
Segatron-XL large + HCP 2022.03	16.1	-	-	-	-	-	-
Transformer-XL + SGD dynamic eval 2019.04	16.3	-	-	-	-	-	-
MEGA 2023.11	17.17	-	-	-	-	-	-
Transformer-XL* 2019.04	17.3	-	-	-	-	-	-
ShortFormer 2023.11	17.47	-	-	-	-	-	-
Shortformer 2022.02	17.5	-	-	-	-	-	-
Transformer-XL (ours) 2022.02	17.58	-	-	-	-	-	-
Perceiver AR 2022.02	17.58	-	-	-	-	-	-
Perceiver AR 2022.02	17.6	-	-	-	-	-	-
Perceiver AR 2022.02	17.66	-	-	-	-	-	-
SEQUENCE 2021.04	17.71	-	-	-	-	-	-
CODA 2021.05	17.81	-	-	-	-	-	-
Perceiver AR 2022.02	17.86	-	-	-	-	-	-
TRANSFORMER+ 2021.05	17.97	-	-	-	-	-	-
Adaptive inputs 2022.02	18	-	-	-	-	-	-
UNCOMPRESSED 2021.04	18.1	-	-	-	-	-	-
DIFFQ 2021.04	18.1	-	-	-	-	-	-
AdaptiveInputs 2023.11	18.15	-	-	-	-	-	-
Tensor-Transformer 2020.03	18.3	-	-	-	-	-	-
Transformer-XL 2022.02	18.3	-	-	-	-	-	-
8 BITS 2021.04	18.3	-	-	-	-	-	-
UNCOMPRESSED 2021.04	18.3	-	-	-	-	-	-
TRANSFORMER 2021.05	18.35	-	-	-	-	-	-
B2T connection 2022.06	18.38	-	-	-	-	-	-
Pre-LN 2022.06	18.53	-	-	-	-	-	-
Baevski and Auli (2019) 2021.04	18.53	-	-	-	-	-	-
GHT 2023.05	18.57	-	-	-	-	-	-
GHT-PS 2023.05	18.58	-	-	-	-	-	-
DIFFQ 2021.04	18.6	-	-	-	-	-	-
QN 2021.04	18.7	-	-	-	-	-	-
LSQ 2021.04	18.9	-	-	-	-	-	-
SEQUENCE 2021.04	18.97	-	-	-	-	-	-
CYCLE 2021.04	19	-	-	-	-	-	-
VT w/ AI 2023.05	19.03	-	-	-	-	-	-
QN 2021.04	19.5	-	-	-	-	-	-
CYCLE (REV) 2021.04	19.6	-	-	-	-	-	-
BERT-Large-CAS 2019.04	19.67	-	-	-	-	-	-
BERT-L-CAS 2023.05	19.67	-	-	-	-	-	-
S4 2023.05	19.69	-	-	-	-	-	-
Tensor-Transformer 2020.03	19.7	-	-	-	-	-	-
QAT 2021.04	19.7	-	-	-	-	-	-
Transformer + adaptive input 2019.04	19.8	-	-	-	-	-	-
Adaptive Input 2020.03	19.8	-	-	-	-	-	-
Post-LN 2022.06	20.24	-	-	-	-	-	-
Vanilla 2021.04	20.39	-	-	-	-	-	-
S4 2023.11	20.47	-	-	-	-	-	-
B2T connection 2022.06	20.5	-	-	-	-	-	-
PQ 2021.04	20.7	-	-	-	-	-	-
Pre-LN 2022.06	20.98	-	-	-	-	-	-
Lite Transformer 2020.04	21.4	-	-	-	-	-	-
Segatron-XL base + HCP 2022.03	21.9	-	-	-	-	-	-
Transformer-XL Base + URPE-based Attention 2022.05	22.4	-	-	-	-	-	-
Segatron-XL base 2022.03	22.5	-	-	-	-	-	-
Tensor-Transformer 2020.03	22.7	-	-	-	-	-	-
Universal 2021.04	22.75	-	-	-	-	-	-
Transformer-XL 2020.03	23.1	-	-	-	-	-	-
Transformer-XL Base 2022.05	23.1	-	-	-	-	-	-
Adaptive Inputs 2020.04	23.2	-	-	-	-	-	-
RFA-Gate-Gaussian 2022.02	23.2	-	-	-	-	-	-
COSFORMER 2022.02	23.5	-	-	-	-	-	-
Tensor-Transformer 2020.03	23.6	-	-	-	-	-	-
Tensor-Transformer 2020.03	23.6	-	-	-	-	-	-
SPADE + SMR 2024.05	23.68	-	-	-	-	-	-
Transformer Large + HCP 2022.03	23.7	-	-	-	-	-	-
Transformer-LS 2024.05	23.71	-	-	-	-	-	-
S6 + SMR 2024.05	23.85	-	-	-	-	-	-
S6 2024.05	23.97	-	-	-	-	-	-
TNN 2023.11	23.98	-	-	-	-	-	-
Transformer Large 2022.03	24	-	-	-	-	-	-
HGRN 2023.11	24.14	-	-	-	-	-	-
SPADE 2024.05	24.18	-	-	-	-	-	-
RWKV 2023.11	24.31	-	-	-	-	-	-
Transformer 2023.11	24.4	-	-	-	-	-	-
Vanilla Transformer 2024.05	24.42	-	-	-	-	-	-
Vanilla Transformer 2022.02	24.5	-	-	-	-	-	-
RFA-Gate-across 2022.02	24.8	-	-	-	-	-	-
Mega + SMR 2024.05	25.28	-	-	-	-	-	-
RFA-Gaussian 2022.02	25.8	-	-	-	-	-	-
FLASH 2023.11	25.92	-	-	-	-	-	-
Mega 2024.05	26.3	-	-	-	-	-	-
RFA-across 2022.02	26.4	-	-	-	-	-	-
cosFormer 2023.11	26.53	-	-	-	-	-	-
4 layer QRNN + Ours 2019.06	27.2	-	-	-	-	-	-
1+elu 2023.11	27.44	-	-	-	-	-	-
gMLP 2023.11	28.08	-	-	-	-	-	-
Tensor-Transformer 2020.03	28.4	-	-	-	-	-	-
Linear Transformer 2022.02	28.7	-	-	-	-	-	-
LSTM 2018.03	29	-	-	-	-	-	-
LSTM + Hebbian + Cache + MbPA 2019.06	29	-	-	-	-	-	-
LSTM+Hebb. 2019.11	29	-	-	-	-	-	-
AWD-LSTM-MoS 2020.03	29	-	-	-	-	-	-
Transformer base + HCP 2022.03	29.1	-	-	-	-	-	-
Transformer base 2022.03	29.2	-	-	-	-	-	-

Showing 100 of 180 rows