Share your thoughts, 1 month free Claude Pro on usSee more

Code Generation on HumanEval+ augmented (test)

90Pass@1

Mercury Coder

Updated 1mo ago

Evaluation Results

Method	Links
Mercury Coder 2026.01		90
Gemini Diffusion 2026.01		89.6
LLaDA2.0-flash 2026.01		87.8
Qwen2.5-Coder-14B-Instruct 2026.01		87.2
Qwen2.5-Coder-32B-Instruct 2026.01		87.2
Qwen2.5-Coder-7B-Instruct 2026.01		84.1
Seed-Diffusion-Preview(0705) 2026.01		82.8
Stable-DiffCoder-8B-Instruct 2026.01		82.3
DeepSeek-Coder-V2-Instruct 2026.01		82.3
GPT-4-Turbo 2024.10		81.7
Qwen3-8B 2026.01		80.5
LLaDA2.0-mini 2026.01		79.9
CodeQwen1.5-7B-Chat 2026.01		78.7
OpenCoder-8B-Instruct 2026.01		78.7
Seed-Coder-8B-Instruct 2026.01		78.7
CodeQwen1.5-7B-Chat 2024.10		77.7
DeepSeek-Coder-V2-Lite-Instruct 2026.01		75.6
DeepSeekCoder-33B-Instruct 2024.10		75
DeepSeek-Coder-33B-Instruct 2026.01		75
Yi-Coder-9B-Chat 2026.01		74.4
OpenCodeInterpreter-DS-33B 2024.10		73.8
WeDLM-8B-Chat 2026.01		73.8
SDLM-32B 2026.01		73.8
WizardCoder-33B-v1.1 2024.10		73.2
Codestral-22B 2026.01		73.2
RMI Top 50% 2026.03		73.17
Full Data 2026.03		72.56
RMI 50-75% 2026.03		72.56
RDS+ 2026.03		71.34
DeepSeek-Coder-6.7B-Instruct 2026.01		71.3
SCAR 2026.03		70.73
Llama3-70B-Instruct 2024.10		70.7
Magicoder-S-DS-6.7B 2024.10		70.7
RMI 50% (G) 2026.03		70.12
RMI Top-25% 2026.03		70.12
Mixtral-8x22B-Instruct-v0.1 2024.10		70.1
Random 2026.03		69.51
RMI QW 50% 2026.03		68.9
RMI Bot-25% 2026.03		67.68
SelfCodeAlign-CQ-7B 2024.10		67.1
IFD 2026.03		66.46
CodeLlama-70B-Instruct 2026.01		65.9
CodeLlama-70B-Instruct 2024.10		65.2
DiffuCoder-7B-Instruct 2026.01		65.2
Snowflake Arctic (480B) 2024.10		64.3
Mistral Large 2024.10		62.8
StarCoder2-15B-Instruct 2026.01		60.4
Llama-3.1-8B-Instruct 2026.01		59.8
Command-R+ (104B) 2024.10		56.7
Gemini Pro 2024.10		55.5
CodeQwen1.5-7B-OctoPack 2024.10		50.6
CodeQwen1.5-7B-Base 2024.10		45.7
Fast-dLLMv2 2026.01		40.2
Mixtral-8x7B-Instruct-v0.1 2024.10		39.6
StarCoder2-15B 2024.10		37.8
CodeLlama-7B-Instruct 2026.01		33.5
OctoCoder-16B 2024.10		32.9