Tokenization on Tamil Wikipedia freq >= 3 (Full Evaluation)
1,699,970Total Tokens
Sarvam-1 (68K Indic BPE)
Updated 1mo ago
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
2026.03 | 1,699,970 | 3.52 | 23.6 | |
2026.03 | 1,375,778 | 2.85 | - | |
2026.03 | 914,619 | 1.89 | 33.5 | |
2026.03 | 896,595 | 1.86 | 34.8 |