RepoEval
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| API Invocation Completion | RepoEval 1.0 (test) | Exact Match50.13 | 24 | |
| Line Completion | RepoEval 1.0 (test) | Exact Match57.75 | 24 | |
| Repository-level code-completion | RepoEval (test) | Exact Match49.9 | 8 | |
| Function Body Completion | RepoEval Function Body Completion (All) | Pass Rate42.63 | 6 | |
| Function Completion | RepoEval leopard (ai-betty) | Pass@564.99 | 3 | |
| Function Completion | RepoEval deepmind tracr | Pass@556.11 | 3 | |
| Function Completion | RepoEval amazon-science patchcore-inspection | Pass@566.94 | 3 |
Showing 7 of 7 rows