Character-level Language Modeling

Benchmarks

Dataset Name	SOTA Method	Metric
enwik8 (test)	Transformer-XL + RMS dynamic eval + decay	BPC0.94	195	3mo ago
text8 (test)	Transformer-XL + RMS dynamic eval + decay	BPC1.038	128	3mo ago
Penn Treebank (test)	3 layer LSTM	BPC1.175	113	3mo ago
Shakespeare modern	SGHMC	Accuracy55.63	48	3mo ago
Hutter Prize Wikipedia (test)	mLSTM	Bits/Char1.08	28	3mo ago
Shakespeare (val)	Byz-NSGDM	Perplexity10.08	27	2mo ago
Penn Treebank char-level (test)	dense-IndRNN	BPC1.16	25	3mo ago
Tiny Shakespeare (val)	EGA-MORLET	Validation Loss1.355	19	20h ago
Enwik8 (val)	Adaptive Transformer	BPC1.04	17	25d ago
text8		BPC0.98	16	3mo ago
text8 (held-out 1M tokens)	SHARP	BPC2.3	14	21h ago
text8 (dev)	Transformer + adaptive span	BPC1.01	13	3mo ago
Shakespeare (train)	Adam	Accuracy59.8	12	1mo ago
Shakespeare (test)	Adam	Accuracy50.2	12	1mo ago
enwik8 (train)	RWKV-RNN	BPC0.72	12	3mo ago
enwik8 (dev)	Adaptive	BPC1	10	3mo ago
Penn Treebank character-level (val)	LayerNorm HM-LSTM	BPC1.24	10	11d ago
text8 (most recent 1M tokens)	SHARP	BPC2.23	7	21h ago
text8 100M regime Backward stream	Transformer (ctx=1024)	Backward BPC2.17	7	21h ago
text8 100M regime (Current stream split)	Transformer (ctx=1024)	Current BPC2.12	7	21h ago
text8 100M regime (Forward split)	Transformer (ctx=1024)	Forward BPC2.19	7	21h ago
Shakespeare	MINGRU + αCMRU	Cross-entropy Loss1.441	6	20d ago
Billion Words (test)	DGflow	JS Divergence (Context 4)0.186	4	3mo ago
Character-level domain-switching dataset	β-MoE	BPC (all)1.659	3	29d ago
WikiText-2 (val)	Transformer	PPL (Validation)3.49	3	3mo ago

Showing 25 of 29 rows