Tokenization on WikiText-103
1.92Latency (ms)
tiktoken
Updated 1mo ago
Evaluation Results
| Method | Links | |
|---|---|---|
2026.03 | 1.92 | |
2026.03 | 4.21 | |
2026.03 | 4.98 | |
2026.03 | 5.77 | |
2026.03 | 6.32 | |
2026.03 | 9.42 | |
2026.03 | 10.9 | |
2026.03 | 12.7 | |
2026.03 | 12.84 | |
2026.03 | 14.5 | |
2026.03 | 18.3 | |
2026.03 | 22.1 | |
2026.03 | 24.1 | |
2026.03 | 29.5 | |
2026.03 | 48.5 | |
2026.03 | 49.4 | |
2026.03 | 53.4 | |
2026.03 | 89.3 | |
2026.03 | 91.3 | |
2026.03 | 189.2 | |
2026.03 | 201 | |
2026.03 | 598.1 | |
2026.03 | 749.9 | |
2026.03 | 1,673 | |
2026.03 | 6,143 |