Share your thoughts, 1 month free Claude Pro on us
See more
Home
/
Benchmarks
Tokenization Efficiency on FLORES-200 (dev+devtest)
Loading...
10.43
Mean Brahmic Fertility
Llama-3.1-8B
1.8084
4.0467
6.285
8.5233
May 28, 2026
Mean Brahmic Fertility
Updated 5d ago
Evaluation Results
Method
Method
Links
Mean Brahmic Fertility
Llama-3.1-8B
Vocab size=128K
2026.05
10.43
Qwen3-8B
Vocab size=152K
2026.05
9.36
IndicBERTv2-SS
Vocab size=200K
2026.05
5.23
DeepSeek-R1
Vocab size=129K
2026.05
5.1
Krutrim-1
Vocab size=70K
2026.05
5.03
Tekken/Sarvam-m
Vocab size=131K
2026.05
4.87
GPT-OSS-120B
Vocab size=200K
2026.05
3.1
BrahmicTokenizer-131K
Vocab size=131K
2026.05
2.84
Gemma-3-1B
Vocab size=262K
2026.05
2.7
Sarvam-1
Vocab size=68K
2026.05
2.3
Sarvam-30B
Vocab size=262K
2026.05
2.14
Feedback
Search any
task
Search any
task