Coding
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Generation | Coding domain | Average Wall-Clock Time (s)4.14 | 40 | |
| Code Generation | Coding | Pass@156.1 | 40 | |
| Code Generation | Coding Eval+ LiveCode (test) | Eval+ Score87.2 | 32 | |
| Response correctness and completeness evaluation | Coding | F1 Score85 | 32 | |
| Pairwise Comparison Accuracy Refinement | Coding | Original Accuracy63.92 | 30 | |
| Coding | Coding Real-data 20% verified | Original Accuracy63.92 | 30 | |
| Multi-Agent System Performance | Coding | TS Score65 | 16 | |
| Coding | Coding (val) | Pass@16100 | 16 | |
| Reasoning | Coding | Normalized Score102.1 | 9 | |
| Prompt Injection Detection | Coding Direct Prompt Injection | FPR0 | 7 | |
| LLM-as-a-judge Evaluation | Coding Qwen (n=8884) | Adjusted Accuracy77.74 | 5 | |
| LLM-as-a-judge Evaluation | Coding GPT-5.4 | Adjusted Accuracy67.2 | 5 | |
| Code Generation | Coding Gender (test) | Cor (%)40 | 5 | |
| Code Generation | Coding Race (test) | Correctness Rate57 | 5 | |
| Coding | Coding Grouped General-Purpose Metrics | Coding Accuracy83.72 | 3 | |
| Prompt Ambiguity Localization | Coding Synthetic (held-out evaluation set) | AUROC0.922 | 3 | |
| Agentic Coding | Coding unseen tasks (test) | Pass@129.2 | 3 | |
| Agent task performance | Coding 20-task (held-out) | G Score81.2 | 2 | |
| Coding | Coding Hard | Baseline Score36.67 | 1 | |
| Coding | Coding Medium | Baseline Score31.96 | 1 | |
| Coding | Coding Easy | Baseline51.14 | 1 |
Showing 21 of 21 rows