Masked Language Modeling

Benchmarks

Dataset Name	SOTA Method	Metric
C4 (val)	FLASH-Quad	PPLX3.828	35	3mo ago
XSUM randomly sampled	MLM	U-PPL3.8	20	3mo ago
SNLI (randomly sampled)	AG	PPL (U)8.57	20	3mo ago
Wikipedia + BookCorpus (dev)	RealFormer	MLM Accuracy74.76	12	3mo ago
Books, CC-News, Stories, Wikipedia (held-out set)	BIGBIRD-ETC	BPC1.274	8	3mo ago
Turkish Datasets (blackerx/turkish_v2, fthbrmnby/turkish_product_reviews, hazal/Turkish-Biomedical-corpus-trM, newmindai/EuroHPC-Legal) (test)	boun-tabilab/TabiBERT	MLM Avg (%)69.57	7	3mo ago
BERT Pretraining Corpus	gMLP_xlarge	Perplexity2.89	7	3mo ago
WikiText-103 raw-v1 (test)	DiBARD	Loss2.5869	6	26d ago
WikiText-103 (train)	GEM (N = 1)	Training Loss6.6188	6	1mo ago
WikiText-103 (val)	GEM (N = 1)	Validation Loss6.6764	6	1mo ago
BERT large	DynamiQ	vNMSE0.0022	6	3mo ago
Ciao (test)	FT(BERT(T2), Manual)	Perplexity5.813	6	3mo ago
ArXiv (test)	FT(BERT(T2), Manual)	Perplexity3.499	6	3mo ago
Reddit (test)	FT(BERT(T2), Manual)	Perplexity8.906	6	3mo ago
Masked LM	KnowBert-W+W	PPL3.5	5	3mo ago
omg prot50 (val)	MUD1	Wall-clock Time (17.5 Target PPL)78	4	2mo ago
6 languages Averaged (test)	NoOverlap	MRR42.7	4	3mo ago
C4	Primer-EZ Decoder	Log Perplexity1.787	4	3mo ago
20 languages	Unigram	MRR52.6	3	3mo ago
GRCh37 human reference genome (held-out set)	BIGBIRD	BPC1.12	3	3mo ago
BLLIP (test)	Transformer	Perplexity101.91	2	3mo ago
PTB (test)	Transformer	Perplexity58.43	2	3mo ago
BERT MLM small (val)	SE-GEM	Validation Loss6.9412	1	1mo ago

Showing 23 of 23 rows