Document Deduplication on RedPajama 100GB web content
94SA
SemHash-LLM
Updated 23d ago
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
2026.07 | 94 | 91 | 89 | 92 | 88 | 91.05 | |
2026.07 | 89 | 85 | 82 | 88 | 83 | 85.18 | |
2026.07 | 88 | 83 | 81 | 76 | 71 | 81.2 | |
2026.07 | 86 | 82 | 80 | 85 | 79 | 84.56 | |
2026.07 | 85 | 79 | 74 | 68 | 55 | 73.45 | |
2026.07 | 83 | 80 | 79 | 74 | 70 | 79.32 | |
2026.07 | 82 | 78 | 68 | 45 | 22 | 61.35 | |
2026.07 | 71 | 65 | 42 | 31 | 18 | 48.2 |