MBPP
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Code Generation | MBPP (test) | Pass@195.1 | 411 | |
| Code Generation | MBPP+ | Accuracy85.75 | 243 | |
| Code Generation | MBPP+ | Pass@184.39 | 238 | |
| Code Generation | MBPP | Pass@192.4 | 211 | |
| Code Generation | MBPP | Pass@189.1 | 193 | |
| Coding | MBPP | Accuracy98.4 | 175 | |
| Code Generation | MBPP | Accuracy79.8 | 165 | |
| Code Generation | MBPP | Accuracy (%)92.2 | 146 | |
| Coding | MBPP+ | Pass@197.88 | 117 | |
| Code Generation | MBPP | Average Acceptance Length (τ)8.99 | 95 | |
| Code Generation | MBPP Plus (test) | Accuracy83.6 | 93 | |
| Code Generation | MBPP-ET | Pass@191.8 | 91 | |
| Code Generation | MBPP | Accuracy96.6 | 90 | |
| Code Generation | MBPP | Accuracy90.5 | 89 | |
| Code Generating | MBPP | Pass@183.1 | 88 | |
| Code Generation | MBPP | Speedup7.68 | 87 | |
| Uncertainty Estimation | MBPP | AUROC0.791 | 79 | |
| Coding | MBPP | Pass@1 Accuracy95.33 | 78 | |
| Code | MBPP | Pass@191.05 | 73 | |
| Code Generation | MBPP Code | Performance (%)83 | 60 | |
| Code Generation | MBPP | Pass@1 Accuracy94.2 | 59 | |
| Code generation | MBPP | Pass@180.4 | 58 | |
| Code Generation | MBPP 3-shot | Accuracy59.6 | 57 | |
| Function-level Code Generation | MBPP+ augmented (test) | Pass@179.6 | 56 | |
| Code Generation | MBPP | Pass@1 Accuracy84.6 | 55 |
Showing 25 of 311 rows
...