Tokenisation on Wikipedia/OpenWebText
99.94F1 Score
Our (VnCoreNLP + BPE)
Updated 1mo ago
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
2026.03 | 99.94 | 99.94 | 75,500 | |
2026.03 | 99.93 | 99.93 | 18,870 | |
2026.03 | 99.9 | 99.9 | 16,670 | |
2026.03 | 99.9 | 99.9 | 90,900 | |
2026.03 | 99.86 | 99.86 | 76,000 | |
2026.03 | 96.5 | 96.5 | 72,400 | |
| 96 | 96 | 68,200 | ||
2026.03 | 95.8 | 95.8 | 70,500 | |
2026.03 | 95.2 | 95.2 | 65,000 |