| Dataset Name | SOTA Method | Metric | Trend | ||
|---|---|---|---|---|---|
| CodeXGLUE | GALLa | Repair Rate15.7 | 21 | 5mo ago | |
| CodeFix | GPT-4o | Fix Success Rate85.5 | 20 | 5mo ago | |
| HumanEvalFix (test) | WaveCoder-Pro-6.7B | Success Rate (Python)59.1 | 19 | 1mo ago | |
| Tufano Medium Abstract 2019 (test) | CoTexT | Top-1 Exact Match Accuracy15.36 | 6 | 5mo ago | |
| Tufano Small Abstract 2019 (test) | NSEdit | Top-1 Accuracy24.04 | 6 | 5mo ago | |
| SWE-bench Lite | ADARUBRIC-DA | r0.77 | 5 | 4mo ago | |
| DeepFix | PaLM-Coder | Pass@182.1 | 5 | 4mo ago | |
| MBPP (test) | CausalFlow | Pass Count523 | 4 | 2mo ago | |
| R4 | EliBadCode | ASR5.34 | 3 | 1mo ago | |
| R3 | EliBadCode | ASR1.92 | 3 | 1mo ago | |
| R2 | ScanNBT | ASR6.77 | 3 | 1mo ago | |
| R1 | EliBadCode | ASR12.21 | 3 | 1mo ago | |
| Tufano Medium Concrete 2019 (test) | NSEdit | Top-1 Acc13.46 | 2 | 5mo ago | |
| Tufano Small Concrete 2019 (test) | NSEdit | Top-1 Acc23.86 | 2 | 5mo ago | |
| DeepFix (test) | - | Pass@1- | 0 | 5mo ago |
