Share your thoughts, 1 month free Claude Pro on us
See more
Home
/
Benchmarks
Tokenization Speedup Comparison on WikiText-103
Loading...
115
Speedup Ratio (GPU-O vs CPU-BPE)
GPUTOK
-1.9168
28.4366
58.79
89.1434
Mar 3, 2026
Speedup Ratio (GPU-O vs CPU-BPE)
Speedup Ratio (GPU-O vs tiktoken)
Speedup Ratio (GPU-O vs HF GPT2)
Updated 1mo ago
Evaluation Results
Method
Method
Links
Speedup Ratio (GPU-O vs CPU-BPE)
Speedup Ratio (GPU-O vs tiktoken)
Speedup Ratio (GPU-O vs HF GPT2)
GPUTOK
Sequence Length=131K
2026.03
115
1.67
31.3
GPUTOK
Sequence Length=16K
2026.03
31.1
3.79
8.34
GPUTOK
Sequence Length=4K
2026.03
13
1.53
3.41
GPUTOK
Sequence Length=1K
2026.03
5.15
0.67
1.35
GPUTOK
Sequence Length=256
2026.03
2.58
0.38
0.91
Feedback
Search any
task
Search any
task
