Share your thoughts, 1 month free Claude Pro on us
See more
Home
/
Benchmarks
Tokenization Efficiency on IN22-Gen (test)
Loading...
10.97
Mean Brahmic Fertility
Llama-3.1-8B
2.0052
4.3326
6.66
8.9874
May 28, 2026
Mean Brahmic Fertility
Updated 5d ago
Evaluation Results
Method
Method
Links
Mean Brahmic Fertility
Llama-3.1-8B
Vocab size=128K
2026.05
10.97
Qwen3-8B
Vocab size=152K
2026.05
9.84
IndicBERTv2-SS
Vocab size=200K
2026.05
5.5
DeepSeek-R1
Vocab size=129K
2026.05
5.43
Krutrim-1
Vocab size=70K
2026.05
5.27
Tekken/Sarvam-m
Vocab size=131K
2026.05
5.21
GPT-OSS-120B
Vocab size=200K
2026.05
3.37
BrahmicTokenizer-131K
Vocab size=131K
2026.05
3.08
Gemma-3-1B
Vocab size=262K
2026.05
2.92
Sarvam-1
Vocab size=68K
2026.05
2.57
Sarvam-30B
Vocab size=262K
2026.05
2.35
Feedback
Search any
task
Search any
task