Long-context evaluation on RULER 128k
98Query Metric (MQ)
Llama-3.1-8B
Updated 4d ago
Evaluation Results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
2025.05 | 98 | 100 | 100 | 100 | 100 | 88 | 64 | 96 | 95.2 | 1.6 | 66.6 | 64 | 36 | 77.6 | - | |
2025.05 | 98 | 92 | 92 | 100 | 100 | 64 | 0 | 93 | 67.2 | 0.4 | 16 | 60 | 40 | 63.3 | - | |
2025.05 | 97 | 100 | 92 | 100 | 100 | 64 | 16 | 96 | 85.6 | 0.4 | 40 | 64 | 36 | 68.5 | - | |
2025.05 | 84 | 100 | 100 | 100 | 96 | 80 | 28 | 69 | 10.4 | 3.6 | 89.3 | 36 | 36 | 64 | - | |
2025.05 | 80 | 100 | 100 | 96 | 96 | 76 | 20 | 55 | 48.8 | 24 | 41.3 | 36 | 32 | 61.9 | - | |
2025.05 | 79.6 | 100 | 100 | 96 | 96 | 72 | 20 | 44 | 19.2 | 23.2 | 44 | 40 | 32 | 58.9 | - | |
2025.05 | 71 | 80 | 68 | 0 | 88 | 48 | 0 | 66 | 59.2 | 0.4 | 52 | 48 | 28 | 46.8 | - | |
2025.05 | 67 | 100 | 100 | 100 | 84 | 72 | 4 | 30 | 4.8 | 1.2 | 100 | 32 | 24 | 55.3 | - | |
2025.05 | 48 | 100 | 100 | 100 | 84 | 28 | 4 | 21 | 14.4 | 1.2 | 60 | 32 | 20 | 47.1 | - | |
2025.05 | 47 | 100 | 100 | 100 | 84 | 16 | 0 | 25 | 3.2 | 0.8 | 61.3 | 32 | 22.7 | 45.5 | - | |
2026.02 | 45.59 | - | - | - | - | - | - | - | 80.2 | - | - | - | - | 61.47 | 58.63 | |
2025.05 | 38 | 100 | 84 | 0 | 84 | 24 | 0 | 19 | 65.6 | 0 | 28 | 48 | 32 | 40.2 | - | |
2026.02 | 35 | - | - | - | - | - | - | - | 90.84 | - | - | - | - | 60.71 | 56.28 | |
2026.02 | 30.5 | - | - | - | - | - | - | - | 51.96 | - | - | - | - | 48.03 | 61.64 | |
2026.02 | 30.4 | - | - | - | - | - | - | - | 76.56 | - | - | - | - | 54.87 | 57.64 | |
2026.02 | 28.7 | - | - | - | - | - | - | - | 90.48 | - | - | - | - | 57.92 | 54.57 | |
2025.05 | 28 | 100 | 84 | 4 | 72 | 24 | 0 | 28 | 16.8 | 0.8 | 69.3 | 24 | 32 | 37.1 | - | |
2026.02 | 27.2 | - | - | - | - | - | - | - | 91.44 | - | - | - | - | 57.22 | 53.02 | |
2025.05 | 23 | 96 | 72 | 0 | 72 | 20 | 0 | 17 | 6.4 | 0.8 | 49.3 | 16 | 8 | 29.2 | - | |
2026.02 | 17.6 | - | - | - | - | - | - | - | 47.68 | - | - | - | - | 39.05 | 51.88 | |
2026.02 | 14 | - | - | - | - | - | - | - | 45.64 | - | - | - | - | 35.46 | 46.74 | |
2025.05 | 5 | 8 | 16 | 0 | 24 | 4 | 0 | 2 | 4 | 0.4 | 77.3 | 16 | 28 | 14.2 | - | |
2025.05 | 4 | 0 | 4 | 0 | 0 | 4 | 0 | 5 | 0 | 0.4 | 9.3 | 24 | 20 | 5.4 | - | |
2025.05 | 4 | 92 | 12 | 0 | 20 | 4 | 0 | 12 | 7.2 | 2 | 53.3 | 20 | 32 | 19.9 | - | |
2025.05 | 4 | 76 | 0 | 0 | 16 | 0 | 0 | 1 | 8 | 0.8 | 69.3 | 20 | 16 | 16.2 | - | |
2025.05 | 3 | 0 | 8 | 0 | 4 | 8 | 0 | 2 | 0.8 | 0 | 66.6 | 40 | 32 | 12.6 | - | |
2025.05 | 2 | 0 | 8 | 0 | 12 | 4 | 0 | 1 | 0 | 0 | 54.6 | 24 | 28 | 10.2 | - | |
2025.05 | 1.14 | 0 | 4 | 4 | 0 | 0 | 0 | 2 | 0 | 0 | 86.6 | 16 | 24 | 10.6 | - | |
2025.05 | 1 | 0 | 4 | 4 | 0 | 0 | 0 | 2 | 0 | 0.8 | 68 | 20 | 16 | 8.9 | - |