DevEval
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Developer Knowledge Evaluation | DevEval | Win Rate61 | 7 | |
| Docstring Evaluation | DevEval 183 human-written docstrings | Score4.938 | 5 | |
| Agentic Coding | DevEval | Solve Rate94.8 | 4 | |
| Repository-level code generation | DevEval | Inference Time442 | 4 | |
| Terminal-related CLI agent task | DevEval | Accuracy39.74 | 2 |
Showing 5 of 5 rows