RepoBench
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Code Completion | RepoBench-P LongBench | Pass@152.05 | 30 | |
| Long-context language generation | RepoBench-P | Average Acceptance Length4.46 | 25 | |
| Code Completion | RepoBench-P | Similarity0.7305 | 17 | |
| Code Generation | RepoBench | Speedup3.57 | 12 | |
| Long Code Completion | RepoBench >8k | Edit Sim51.24 | 12 | |
| Long Code Completion | RepoBench 4k-8k | Edit Similarity53.3 | 12 | |
| Long Code Completion | RepoBench 0-4k | Edit Similarity52.82 | 12 | |
| Long-context code completion | RepoBench-P | MAT1.83 | 11 | |
| Repository-level code-completion | RepoBench (test) | Exact-match Accuracy65.9 | 7 | |
| Code Generation | RepoBench-P Python, XF-Random | Execution Match (EM)64.5 | 6 | |
| Code Generation | RepoBench-P Python XF-First | Exact Match (EM)52.4 | 6 | |
| Coding | RepoBench | Pass@125.3 | 6 | |
| code generation | RepoBench P | Score15.04 | 5 | |
| Code Completion | RepoBench | Pass@k Score48.92 | 1 |
Showing 14 of 14 rows