SWE-Bench
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Software Engineering Issue Resolution | SWE-bench Verified | Resolution Rate89.2 | 110 | |
| Agentic Coding | SWE-bench Verified | Percentage Resolved87.6 | 71 | |
| Software Engineering Task Resolution | SWE-bench Verified | Resolution Rate73.3 | 63 | |
| Software Engineering | SWE-bench Verified | Success Rate80.9 | 53 | |
| Software Engineering | SWE-bench Verified | Accuracy62.6 | 43 | |
| Automated Software Engineering | SWE-bench Verified | Resolved Rate1,770 | 39 | |
| Software Engineering | SWE-bench Lite | Speedup4.66 | 36 | |
| Issue Resolution | SWE-bench Verified (test) | Pass Rate77.2 | 36 | |
| Software Engineering | SWE-bench verified (All) | Success Rate93.8 | 32 | |
| Software Engineering | SWE-bench Verified | Resolution Rate83.8 | 32 | |
| Software Engineering Agent Task | SWE-Bench Pro | Pass@3100 | 28 | |
| Function-level Code Localization | SWE-bench Live Lite | Acc@174.8 | 25 | |
| File-level Code Localization | SWE-bench Live Lite | Acc@182.1 | 25 | |
| Function-level Code Localization | SWE-bench Verified (Lite) | Acc@183.4 | 25 | |
| File-level Code Localization | SWE-bench Verified Lite | Accuracy@191.9 | 25 | |
| Code Generation | SWE-Bench | Speedup Ratio2.96 | 24 | |
| Code Localization | SWE-bench Verified (test) | File Precision86.38 | 24 | |
| Software Engineering | SWE-Bench Lite | Accuracy57.09 | 23 | |
| Software Engineering | SWE-bench Verified | Pass@172 | 23 | |
| File-level localization | SWE-Bench Verified | Hit@189.6 | 21 | |
| Software Engineering Issue Resolution | SWE-Bench-Pro (test) | Pass@159.9 | 21 | |
| Code Agent | SWE-Bench Verified | Score0.809 | 21 | |
| Agentic Coding | SWE-bench 50 tasks sampled | Resolved Count12 | 20 | |
| Software Engineering | SWE-Bench Verified | Pass Rate72 | 20 | |
| Software Engineering | SWE-Bench Multilingual 1.0 (test) | Resolution Rate75.2 | 20 |
Showing 25 of 160 rows