CRUXEval
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Code Reasoning | CRUXEval | Input-CoT Accuracy98.8 | 56 | |
| Code Output Prediction | CRUXEval-O | Pass@149.8 | 47 | |
| Code Input Prediction | CRUXEval-I | Pass@150 | 47 | |
| Code Reasoning | CRUXEval-O | Accuracy83.5 | 28 | |
| Code Reasoning | CRUXEval | Accuracy68.6 | 21 | |
| Code Reasoning | CruxEval Output | Score51 | 12 | |
| Coding | CRUXEval-O | Score87.5 | 10 | |
| Coding | CRUXEval | Pass@155.9 | 6 | |
| Code Reasoning | CRUXEval I | Accuracy74 | 4 | |
| Code | CruxEval o | Exact Match35.3 | 4 | |
| Code | CruxEval-i | Exact Match36.2 | 4 | |
| Output Prediction | CruxEval | Pass@1 (step_return)77.9 | 3 | |
| Input Prediction | CruxEval | Pass@1 (inv_step_call)66.5 | 3 | |
| Code Reasoning (Output Prediction) | CRUXEval-O 1-shot | Accuracy84.01 | 3 | |
| Code Reasoning (Input Prediction) | CRUXEval-I 1-shot | Accuracy79.75 | 3 |
Showing 15 of 15 rows