PY150
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Next token prediction | py150 | MRR67.2 | 16 | |
| Code Summarization | Py150 (test) | ASR81.36 | 12 | |
| Method Name Prediction | Py150 (test) | ASR77.52 | 12 | |
| Code Summarization | Py150 | Recall100 | 12 | |
| Method Name Prediction | Py150 | Recall99.99 | 12 | |
| Code Completion | py150 OOD WILDS (test) | MethodClass Accuracy68.3 | 7 | |
| Line-level code completion | PY150 | EM44.86 | 7 | |
| Code Generation | Py150 | Accuracy10.57 | 4 | |
| Code Modeling | PY150 (test) | Accuracy62.1 | 2 |
Showing 9 of 9 rows