Intrinsic tokenization quality evaluation on CulturaX 18 languages sample

4.89WF

Text

Updated 1mo ago

Evaluation Results

Method	Links
Text 2026.06		4.89	92	2.02	13	8.81	6.37	3.99	2.26	2	0.29	3.3	36
IPA 2026.06		3.18	90	2.72	16	9.14	6.06	3.7	2.98	2	0.16	1.72	17