Long-context evaluation on RULER 32K context length (test)
100Niah1 Score
fp16 Baseline
Updated 4d ago
Evaluation Results
| Method | Links | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2024.01 | 100 | 99.8 | 98.6 | 94 | 68.2 | 11 | 55.95 | 64.5 | 37.88 | 9.64 | 30.4 | 31.6 | 31.6 | 56.4 | |
2025.06 | 100 | 95.83 | 52.08 | 87.5 | 54.17 | - | 52.34 | 56.51 | 26.87 | - | 36.36 | 23.96 | 34.38 | 56.37 | |
2025.06 | 100 | 98.96 | 83.33 | 93.75 | 78.12 | - | 65.62 | 54.17 | 20 | - | 43.4 | 28.12 | 37.5 | 63.91 | |
2024.01 | 99.8 | 98.8 | 95.2 | 92.8 | 61.6 | 6.4 | 47.5 | 54.45 | 41.04 | 8.52 | 29.33 | 31 | 31 | 53.65 | |
2025.06 | 97.92 | 93.75 | 54.17 | 83.33 | 71.88 | - | 59.38 | 43.49 | 16.04 | - | 51.39 | 30.21 | 35.42 | 57.91 | |
2024.01 | 95.4 | 86.8 | 49.8 | 73.6 | 23.4 | 0 | 16.65 | 22.95 | 22.52 | 5.14 | 24 | 26.4 | 28.4 | 36.54 | |
2025.06 | 93.75 | 100 | 91.67 | 93.75 | 81.25 | - | 66.67 | 52.08 | 21.04 | - | 48.61 | 30.21 | 36.46 | 65.04 | |
2024.01 | 76 | 85.6 | 59.6 | 72.6 | 11.4 | 0 | 34.7 | 46.45 | 39.6 | 8.26 | 30.53 | 24.8 | 27.6 | 39.78 | |
2025.06 | 25 | 2.08 | 0 | 0 | 0 | - | 0 | 0 | 1.04 | - | 19.44 | 14.58 | 16.67 | 7.16 | |
2025.06 | 25 | 1.04 | 0 | 0 | 0 | - | 0.26 | 0 | 3.96 | - | 20.49 | 22.92 | 26.04 | 9.25 | |
2025.06 | 1.04 | 5.21 | 1.04 | 6.25 | 1.04 | - | 6.25 | 4.17 | 0.21 | - | 54.86 | 22.92 | 27.08 | 11.82 | |
2025.06 | 0 | 0 | 0 | 3.12 | 0 | - | 0.78 | 0 | 0 | - | 20.49 | 15.62 | 16.67 | 5.15 |