Share your thoughts, 1 month free Claude Pro on usSee more

Code Generation on MBPP (Pass@1)

83.1Pass@1

MapCoder

Updated 1mo ago

Evaluation Results

Method	Links
MapCoder 2024.05		83.1
CoT 2024.05		82.4
Direct 2024.05		81.1
LATS (ReAct) 2023.10		81.1
MapCoder 2024.05		78.3
Reflexion 2024.05		78.3
DeepSeek-R1 2025.12		77.43
Self-Planning 2024.05		75.8
CompassMax-V3-Thinking 2025.12		73.54
DeepSeekCoder-33B 2024.02		73.2
Reflexion 2024.05		73
RAP 2023.10		71.4
Analogical 2024.05		70.5
DeepSeekCoder-6.7B 2024.02		70.2
Reflexion 2023.10		70
Self-collaboration 2024.05		68.2
CompassMax-V3 2025.12		67.7
ReAct 2023.10		67
StarCoder2-15B 2024.02		66.2
ToT 2023.10		65.8
CodeLlama-34B 2024.02		65.4
CodeLlama-13B 2024.02		62.4
Mixtral 8x7B 2024.01		60.7
Analogical 2024.05		58.4
StarCoder2-3B 2024.02		57.4
Self-Planning 2024.05		55.7
DeepSeekCoder-1.3B 2024.02		55.4
Phi-2 2024.07		55
CoT 2023.10		54.9
CoT 2024.05		54.5
StarCoder2-7B 2024.02		54.4
Breadcrumbs 2025.02		53.4
StableCode-3B 2024.02		53.1
CodeLlama-7B 2024.02		52.1
StarCoderBase-15B 2024.02		50.6
Ties-Merging 2025.02		50.2
Mistral 7B 2024.01		50.2
LLaMA 2 70B 2024.01		49.8
Direct 2024.05		49.8
Breadcrumbs 2025.02		49.4
Model Stock 2025.02		47.8
StarCoderBase-7B 2024.02		47.4
LED-Merging 2025.02		47.2
Model Stock 2025.02		47
Llama-3-8B 2024.07		47
DeepSeekMoE Chat 16B 2024.01		46.2
Llama-3-SynE 2024.07		45.6
LED-Merging 2025.02		44.6
StarCoderBase-3B 2024.02		42.6
Ties-Merging 2025.02		41.6
LLaMA 1 33B 2024.01		40.9
DeepSeekMoE 16B 2024.01		39.2
DeepSeekMoE 16B 2024.01		39.2
DeepSeek Chat 7B 2024.01		39
DeepSeek 7B (Dense) 2024.01		39
MAmmoTH2-8B 2024.07		38.8
Task Arithmetic 2025.02		37.8
Qwen2-1.5B 2024.07		37.4
Mistral-7B-v0.3 2024.07		36
LLaMA 2 13B 2024.01		35.4
LED-Merging 2025.02		33.8
w/o Merging 2025.02		33.6
DeepSeek 67B (Dense) 2024.01		33.6
Task Arithmetic 2025.02		33.2
DeepSeekMoE 145B 2024.01		33.2
Ties-Merging 2025.02		33
DCLM-7B 2024.07		32.6
Breadcrumbs 2025.02		32
DeepSeekMoE 142B (Half Activated) 2024.01		32
w/o Merging 2025.02		31.4
Ties-Merging 2025.02		31.4
Gemma-2B 2024.07		29.2
Breadcrumbs 2025.02		28.4
LLaMA2 SFT 7B 2024.01		27.8
GShard 137B 2024.01		27.6
LLaMA 2 7B 2024.01		26.1
Task Arithmetic 2025.02		25.8
LED-Merging 2025.02		23.8
w/o Merging 2025.02		22.82
Qwen2-0.5B 2024.07		22
Task Arithmetic 2025.02		21.8
LLaMA2 7B 2024.01		21.8
Qwen1.5-1.8B 2024.07		18
Llama-3-Chinese-8B 2024.07		14.8
Model Stock 2025.02		14
Model Stock 2025.02		6.2
Galactica-6.7B 2024.07		2
w/o Merging 2025.02		1