Share your thoughts, 1 month free Claude Pro on usSee more

Language Modeling on WikiText-103 (val)

1.01PPL

BiLSTM

Updated 12d ago

Evaluation Results

Method
BiLSTM 2025.12	1.01	-	-	-	0.01	0.02	99
Ensemble of All 2023.11	13.11	-	-	-	-	-	-
kNN LM 2023.11	14.39	-	-	-	-	-	-
Transformer-XL + RMS dynamic eval 2019.04	15.8	-	-	-	-	-	-
Compressive Transformer 2019.11	16	-	-	-	-	-	-
Compressive Transformer 2022.03	16	-	-	-	-	-	-
Compressive Transformer 2022.02	16	-	-	-	-	-	-
Segatron-XL large + HCP 2022.03	16.1	-	-	-	-	-	-
Transformer-XL + SGD dynamic eval 2019.04	16.3	-	-	-	-	-	-
MEGA 2023.11	17.17	-	-	-	-	-	-
Transformer-XL* 2019.04	17.3	-	-	-	-	-	-
ShortFormer 2023.11	17.47	-	-	-	-	-	-
Shortformer 2022.02	17.5	-	-	-	-	-	-
Transformer-XL (ours) 2022.02	17.58	-	-	-	-	-	-
Perceiver AR 2022.02	17.58	-	-	-	-	-	-
Perceiver AR 2022.02	17.6	-	-	-	-	-	-
Perceiver AR 2022.02	17.66	-	-	-	-	-	-
SEQUENCE 2021.04	17.71	-	-	-	-	-	-
CODA 2021.05	17.81	-	-	-	-	-	-
Perceiver AR 2022.02	17.86	-	-	-	-	-	-
TRANSFORMER+ 2021.05	17.97	-	-	-	-	-	-
Adaptive inputs 2022.02	18	-	-	-	-	-	-
Hybrid-GRU 2026.02	18.07	-	-	-	-	-	-
UNCOMPRESSED 2021.04	18.1	-	-	-	-	-	-
DIFFQ 2021.04	18.1	-	-	-	-	-	-
AdaptiveInputs 2023.11	18.15	-	-	-	-	-	-
suGRU(100M) 2026.02	18.29	-	-	-	-	-	-
Tensor-Transformer 2020.03	18.3	-	-	-	-	-	-
Transformer-XL 2022.02	18.3	-	-	-	-	-	-
8 BITS 2021.04	18.3	-	-	-	-	-	-
UNCOMPRESSED 2021.04	18.3	-	-	-	-	-	-
Transformer 2026.02	18.32	-	-	-	-	-	-
TRANSFORMER 2021.05	18.35	-	-	-	-	-	-
B2T connection 2022.06	18.38	-	-	-	-	-	-
Pre-LN 2022.06	18.53	-	-	-	-	-	-
Baevski and Auli (2019) 2021.04	18.53	-	-	-	-	-	-
GHT 2023.05	18.57	-	-	-	-	-	-
GHT-PS 2023.05	18.58	-	-	-	-	-	-
DIFFQ 2021.04	18.6	-	-	-	-	-	-
QN 2021.04	18.7	-	-	-	-	-	-
LSQ 2021.04	18.9	-	-	-	-	-	-
Diff Attention 2026.04	18.93	-	-	-	-	-	-
SEQUENCE 2021.04	18.97	-	-	-	-	-	-
CYCLE 2021.04	19	-	-	-	-	-	-
VT w/ AI 2023.05	19.03	-	-	-	-	-	-
suGRU 2026.02	19.28	-	-	-	-	-	-
Coupled QK 2026.04	19.35	-	-	-	-	-	-
QN 2021.04	19.5	-	-	-	-	-	-
Standard (baseline) 2026.04	19.55	-	-	-	-	-	-
CYCLE (REV) 2021.04	19.6	-	-	-	-	-	-
BERT-Large-CAS 2019.04	19.67	-	-	-	-	-	-
BERT-L-CAS 2023.05	19.67	-	-	-	-	-	-
S4 2023.05	19.69	-	-	-	-	-	-
Tensor-Transformer 2020.03	19.7	-	-	-	-	-	-
QAT 2021.04	19.7	-	-	-	-	-	-
Transformer + adaptive input 2019.04	19.8	-	-	-	-	-	-
Adaptive Input 2020.03	19.8	-	-	-	-	-	-
Coupled QK 2026.04	20.12	-	-	-	-	-	-
Post-LN 2022.06	20.24	-	-	-	-	-	-
Vanilla 2021.04	20.39	-	-	-	-	-	-
S4 2023.11	20.47	-	-	-	-	-	-
B2T connection 2022.06	20.5	-	-	-	-	-	-
PQ 2021.04	20.7	-	-	-	-	-	-
Pre-LN 2022.06	20.98	-	-	-	-	-	-
Diff Attention 2026.04	21.11	-	-	-	-	-	-
Lite Transformer 2020.04	21.4	-	-	-	-	-	-
Standard (baseline) 2026.04	21.57	-	-	-	-	-	-
GQA (baseline) 2026.04	21.87	-	-	-	-	-	-
Segatron-XL base + HCP 2022.03	21.9	-	-	-	-	-	-
Transformer-XL Base + URPE-based Attention 2022.05	22.4	-	-	-	-	-	-
Transformer-XL 2026.03	22.436	-	-	-	-	-	-
Segatron-XL base 2022.03	22.5	-	-	-	-	-	-
Mamba 2026.02	22.58	-	-	-	-	-	-
Transformer-XL 2026.03	22.65	-	-	-	-	-	-
Tensor-Transformer 2020.03	22.7	-	-	-	-	-	-
Universal 2021.04	22.75	-	-	-	-	-	-
Transformer-XL 2020.03	23.1	-	-	-	-	-	-
Transformer-XL Base 2022.05	23.1	-	-	-	-	-	-
HGRN2 2026.02	23.1	-	-	-	-	-	-
Adaptive Inputs 2020.04	23.2	-	-	-	-	-	-
RFA-Gate-Gaussian 2022.02	23.2	-	-	-	-	-	-
COSFORMER 2022.02	23.5	-	-	-	-	-	-
Tensor-Transformer 2020.03	23.6	-	-	-	-	-	-
Tensor-Transformer 2020.03	23.6	-	-	-	-	-	-
SPADE + SMR 2024.05	23.68	-	-	-	-	-	-
Transformer Large + HCP 2022.03	23.7	-	-	-	-	-	-
Transformer-LS 2024.05	23.71	-	-	-	-	-	-
S6 + SMR 2024.05	23.85	-	-	-	-	-	-
S6 2024.05	23.97	-	-	-	-	-	-
TNN 2023.11	23.98	-	-	-	-	-	-
Transformer Large 2022.03	24	-	-	-	-	-	-
HGRN 2023.11	24.14	-	-	-	-	-	-
SPADE 2024.05	24.18	-	-	-	-	-	-
RWKV 2023.11	24.31	-	-	-	-	-	-
RWKV-4 2026.02	24.31	-	-	-	-	-	-
Transformer 2023.11	24.4	-	-	-	-	-	-
Transformer 2026.02	24.4	-	-	-	-	-	-
Vanilla Transformer 2024.05	24.42	-	-	-	-	-	-
Vanilla Transformer 2022.02	24.5	-	-	-	-	-	-
RFA-Gate-across 2022.02	24.8	-	-	-	-	-	-

Showing 100 of 214 rows