Vulnerability Reproduction on CyberGym (evaluation)
1,600Total Rollouts
GPT-5.4 mini Best-of-8
Updated 23d ago
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
2026.07 | 1,600 | 7 | 93 | 2 | 13 | 85 | 42.5 | 35.9 | |
| 1,600 | 9 | 100 | 0 | 5 | 86 | 43 | 36.3 | ||
2026.07 | 1,600 | 4 | 49 | 1 | 20 | 126 | 63 | 56.1 | |
2026.07 | 1,600 | 6 | 35 | 1 | 17 | 141 | 70.5 | 63.8 | |
2026.07 | 1,600 | 6 | 77 | 1 | 7 | 109 | 54.5 | 47.6 | |
2026.07 | 1,600 | 2 | 62 | 2 | 12 | 122 | 61 | 54.1 | |
| 1,026 | 0 | 86 | 0 | 8 | 106 | 53 | 46.1 | ||
2026.07 | 924 | 0 | 99 | 0 | 11 | 90 | 45 | 38.3 | |
| 891 | 2 | 26 | 40 | 12 | 120 | 60 | 53.1 | ||
| 842 | 0 | 56 | 2 | 8 | 134 | 67 | 60.2 | ||
| 753 | 0 | 28 | 0 | 18 | 154 | 77 | 70.7 | ||
| 750 | 0 | 51 | 1 | 31 | 117 | 58.5 | 51.6 | ||
| 719 | 0 | 21 | 19 | 18 | 142 | 71 | 64.4 | ||
| 619 | 1 | 34 | 0 | 20 | 145 | 72.5 | 65.9 | ||
| 560 | 1 | 10 | 15 | 5 | 169 | 84.5 | 78.8 |