Our new X account is live! Follow @wizwand_team for updates
Home
/
Benchmarks
Cross-language Vocabulary Overlap on 6 languages (Same script split)
Loading...
0.62
JSD
Unigram
0.6048
0.7074
0.81
0.9126
May 26, 2023
JSD
Updated 4d ago
Evaluation Results
Method
Method
Links
JSD
Unigram
Tokenizer=Unigram
2023.05
0.62
TokMix
Tokenizer=TokMix
2023.05
0.65
BPE
Tokenizer=BPE
2023.05
0.68
NoOverlap
Tokenizer=NoOverlap
2023.05
1
Feedback
Search any
task
Search any
task