MBPP
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Code Generation | MBPP (test) | Pass@195.1 | 405 | |
| Code Generation | MBPP+ | Pass@184.39 | 238 | |
| Code Generation | MBPP+ | Accuracy85.75 | 236 | |
| Code Generation | MBPP | Pass@192.4 | 211 | |
| Code Generation | MBPP | Pass@189.1 | 193 | |
| Code Generation | MBPP | Accuracy79.8 | 165 | |
| Code Generation | MBPP | Accuracy (%)92.2 | 146 | |
| Coding | MBPP | Accuracy98.4 | 145 | |
| Coding | MBPP+ | Pass@197.88 | 117 | |
| Code Generation | MBPP-ET | Pass@191.8 | 91 | |
| Code Generation | MBPP | Accuracy96.6 | 90 | |
| Code Generation | MBPP | Accuracy90.5 | 89 | |
| Code Generation | MBPP Plus (test) | Accuracy83.6 | 89 | |
| Code Generating | MBPP | Pass@183.1 | 88 | |
| Code Generation | MBPP | Speedup7.68 | 79 | |
| Coding | MBPP | Pass@1 Accuracy95.33 | 78 | |
| Code | MBPP | Pass@191.05 | 73 | |
| Code Generation | MBPP Code | Performance (%)83 | 60 | |
| Code Generation | MBPP | Pass@1 Accuracy94.2 | 59 | |
| Code generation | MBPP | Pass@180.4 | 58 | |
| Function-level Code Generation | MBPP+ augmented (test) | Pass@179.6 | 56 | |
| Code Generation | MBPP Sanitized | Accuracy85.7 | 51 | |
| Code Generation | MBPP | TPS4,290 | 50 | |
| Code Generation | MBPP+ | Score94.2 | 43 | |
| Code Generation | MBPP+ | Pass@173.75 | 40 |
Showing 25 of 274 rows
...