Code Generation on BigCodeBench-Completion Hard

36.5pass@1

GPT-4o-2024-08-06

Updated 1mo ago

Evaluation Results

Method	Links
GPT-4o-2024-08-06 2025.12		36.5
Claude-3.5-Sonnet-20241022 2025.12		35.1
Qwen2.5-Coder-32B-Instruct 2025.12		33.8
DeepSeek-Coder-V2-Instruct 2026.01		33.1
Stable-DiffCoder-8B-Instruct 2026.01		31.8
Qwen2.5-Coder-14B-Instruct 2025.12		29.7
DS-Coder-V2-Instruct 2025.12		29.7
UCoder-32B 2025.12		27.7
Seed-Coder-8B-Instruct 2026.01		26.4
Qwen2.5-Coder-32B 2025.12		26.4
Codestral-22B 2026.01		24.3
UCoder-14B 2025.12		24.3
Qwen3-8B 2026.01		23
UCoder-7B 2025.12		22.3
Qwen2.5-Coder-14B 2025.12		22.3
DeepSeek-Coder-33B-Instruct 2026.01		20.9
Qwen2.5-Coder-32B-Instruct 2026.01		20.9
DS-Coder-33B-Instruct 2025.12		20.9
Qwen2.5-Coder-7B-Instruct 2026.01		20.3
Qwen2.5-Coder-7B-Instruct 2025.12		20.3
OpenCoder-8B-Instruct 2026.01		18.9
OpenCoder-8B-Instruct 2025.12		18.9
DeepSeek-Coder-V2-Lite-Instruct 2026.01		18.2
DS-Coder-V2-Lite-Instruct 2025.12		18.2
Yi-Coder-9B-Chat 2026.01		17.6
Qwen2.5-Coder-14B-Instruct 2026.01		16.2
Qwen2.5-Coder-7B 2025.12		16.2
DeepSeek-Coder-6.7B-Instruct 2026.01		15.5
CodeQwen1.5-7B-Chat 2026.01		15.5
CodeLlama-70B-Instruct 2026.01		15.5
DS-Coder-6.7B-Instruct 2025.12		15.5
StarCoder2-15B-Instruct 2026.01		14.9
Starcoder2-15B-Instruct-v0.1 2025.12		14.9
Llama-3.1-8B-Instruct 2026.01		13.5
CodeLlama-34B-Instruct 2025.12		10.8
CodeLlama-13B-Instruct 2025.12		6.8
CodeLlama-7B-Instruct 2026.01		4.1
CodeLlama-7B-Instruct 2025.12		4.1