SlimPajama
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Language Modeling | SlimPajama | Perplexity (PPL)2.97 | 77 | |
| Language Modeling | SlimPajama latest (val) | Validation Loss3.078 | 26 | |
| Language Modeling | SlimPajama (test) | PPL (CommonCrawl)0.5402 | 25 | |
| Language Modeling | SlimPajama data-restricted | Average Perplexity24.35 | 17 | |
| Language Modeling | SlimPajama (val) | Perplexity3.09 | 13 | |
| Language Modeling | SlimPajama large-scale (train) | L(ψ)2.129 | 8 | |
| Language Modeling | SlimPajama Data Abundant Regime 6B (test) | Arxiv Perplexity11.46 | 6 | |
| Language Modeling | SlimPajama data-restricted regime 6B (test) | Perplexity (Arxiv)16.85 | 6 | |
| Language Modeling | SlimPajama-672B (val) | Validation Perplexity8.09 | 6 | |
| Language Modeling | SlimPajama 6B (val) | Validation Loss2.3 | 4 | |
| Language Modeling | SlimPajama-6B (train) | Train Loss2.1 | 4 | |
| Factuality Evaluation | SlimPajama | Pointwise Score63.5 | 3 | |
| Generation Quality and Coherence Evaluation | SlimPajama Quality Evaluation (test) | Gen Quality (Std. Prefix)86.3 | 3 | |
| Language Modeling | SlimPajama 10M (dev) | Perplexity9.219 | 3 | |
| Language Modeling | SlimPajama 6B tokens | Arxiv Perplexity10.26 | 2 | |
| Self-attention inverse temperature scaling analysis | SlimPajama | Tie Percentage6 | 2 |
Showing 16 of 16 rows