Language Modeling on Fineweb-edu distillation 8B to 300M
2.74LM Loss
Random Sampling KD
Updated 1mo ago
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
2025.03 | 2.74 | 100 | 0.9 | |
2025.03 | 2.75 | 100 | 0.7 | |
2025.03 | 2.77 | 57 | 1.9 | |
2025.03 | 2.78 | -47 | 1.7 | |
2025.03 | 2.8 | 5 | 2.2 | |
2025.03 | 2.81 | 0 | 1.2 | |
2025.03 | 2.85 | -78 | 1.4 |