SOTA Parallel Code Generation on ParEval and PapersWithCode

98Build@1

Gemini-2.5-Pro

Updated 3mo ago

Evaluation Results

Method
Gemini-2.5-Pro 2026.03	98	77	10.4
Gemini-3-Pro 2026.03	97	82	7.42
Qwen2.5-Coder-32B 2026.03	93	11	9.98
DeepSeek-6.7B-Base 2026.03	89	11	3.65
DeepSeek-Syntax 2026.03	85	12	6.6
Gemini-2.5-Parlay 2026.03	84	33	106.87
Qwen2.5-Coder-32B 2026.03	82	45	5.64
DeepSeek-Coder-V2-Lite-Base 2026.03	80	9	2.57
DeepSeek-Parlay 2026.03	79	35	16.4
StarCoder2-15B 2026.03	77	27	3.58
GPT-5 Thinking 2026.03	73	63	14.03
DeepSeek-Coder-V2-Lite-Base 2026.03	73	29	6.26
Qwen3-Rust 2026.03	64	46	6.1
Qwen2.5-Coder-32B-Instruct 2026.03	63	49	5.97
Qwen2.5-Coder-32B-Instruct 2026.03	61	41	12.91
Qwen3-Coder-30B-Instruct 2026.03	61	50	5.7
Gemini-2.5-Flash 2026.03	58	29	13.42
Qwen3-Parlay 2026.03	50	33	8.63
DeepSeek-Coder-V2-Lite-Instruct 2026.03	40	2	0.77
Claude Opus 4.5 2026.03	28	27	0.65
Gemini-3-Pro 2026.03	25	23	12.29