| C4 (val) | FLASH-Quad | PPLX3.828 | | 35 | 4d ago |
| XSUM randomly sampled | MLM | U-PPL3.8 | | 20 | 4d ago |
| SNLI (randomly sampled) | AG | PPL (U)8.57 | | 20 | 4d ago |
| Wikipedia + BookCorpus (dev) | RealFormer | MLM Accuracy74.76 | | 12 | 4d ago |
| Books, CC-News, Stories, Wikipedia (held-out set) | BIGBIRD-ETC | BPC1.274 | | 8 | 4d ago |
| Turkish Datasets (blackerx/turkish_v2, fthbrmnby/turkish_product_reviews, hazal/Turkish-Biomedical-corpus-trM, newmindai/EuroHPC-Legal) (test) | boun-tabilab/TabiBERT | MLM Avg (%)69.57 | | 7 | 4d ago |
| BERT Pretraining Corpus | gMLP_xlarge | Perplexity2.89 | | 7 | 4d ago |
| BERT large | DynamiQ | vNMSE0.0022 | | 6 | 4d ago |
| Ciao (test) | FT(BERT(T2), Manual) | Perplexity5.813 | | 6 | 4d ago |
| ArXiv (test) | FT(BERT(T2), Manual) | Perplexity3.499 | | 6 | 4d ago |
| Reddit (test) | FT(BERT(T2), Manual) | Perplexity8.906 | | 6 | 4d ago |
| Masked LM | KnowBert-W+W | PPL3.5 | | 5 | 4d ago |
| 6 languages Averaged (test) | NoOverlap | MRR42.7 | | 4 | 4d ago |
| C4 | Primer-EZ Decoder | Log Perplexity1.787 | | 4 | 4d ago |
| 20 languages | Unigram | MRR52.6 | | 3 | 4d ago |
| GRCh37 human reference genome (held-out set) | BIGBIRD | BPC1.12 | | 3 | 4d ago |
| BLLIP (test) | Transformer | Perplexity101.91 | | 2 | 4d ago |
| PTB (test) | Transformer | Perplexity58.43 | | 2 | 4d ago |
