Share your thoughts, 1 month free Claude Pro on usSee more

Code Generation on HumanEval and MBPP (Aggregated)

85.6Overall Average Score

GPT-4-1106-preview

Updated 1mo ago

Evaluation Results

Method
GPT-4-1106-preview 2024.05	85.6	-	-	-	-	-	77.5	-	-	-	-
CodeSim 2025.02	82.7	95.1	87.2	90.7	-	-	-	86	89.4	61.2	76
CodeSim 2025.02	81.3	94.5	84.8	89.7	-	-	-	81.7	87	61.5	75.6
GPT-3.5-Turbo-1106 2024.05	79.7	-	-	-	-	-	70.2	-	-	-	-
MapCoder 2025.02	79	90.2	81.7	88.7	-	-	-	80.5	84.1	59.2	74
WizardLM(filtered) 2025.12	78.9	77.4	-	80.4	-	-	-	-	-	-	-
DataFlow-SFT-15K(filtered) 2025.12	78.9	82.9	-	74.9	-	-	-	-	-	-	-
WizardLM(random) 2025.12	78.8	75.6	-	82	-	-	-	-	-	-	-
MapCoder 2025.02	78.6	93.9	83.5	83.1	-	-	-	82.9	86.8	57.7	70.4
CoT 2025.02	78.6	90.9	87.2	82.9	-	-	-	82.3	86.8	57.9	70.4
DataFlow-SFT-15K(random) 2025.12	77.9	79.9	-	75.9	-	-	-	-	-	-	-
Self-Planning 2025.02	77	89	84.1	82.6	-	-	-	80.5	84.5	56.4	69.5
Direct 2025.02	76.5	90.2	82.3	81.1	-	-	-	81.1	84.5	55.9	68.5
Reflexion 2025.02	76	87.2	81.1	81.1	-	-	-	81.1	83.1	56.7	68.9
CodeSim 2025.02	75.1	86	73.2	86.4	-	-	-	72	77.1	59.7	73.1
Alpaca(filtered) 2025.12	74.8	73.8	-	75.7	-	-	-	-	-	-	-
Reflexion 2025.02	74.4	91	81.7	78.3	-	-	-	78.7	83.8	51.9	65.1
CHORD 2025.09	73.76	-	-	-	-	-	-	-	-	-	-
Alpaca(random) 2025.12	73.6	71.3	-	75.9	-	-	-	-	-	-	-
Analogical 2025.02	73.6	88.4	83.5	75.1	-	-	-	80.5	84.1	50.9	63
A-SFT → A-GRPO 2025.09	73.26	-	-	-	-	-	-	-	-	-	-
Direct 2025.02	73.2	80.1	81.7	81.1	-	-	-	73.8	78.5	54.7	67.9
A-GRPO 2025.09	73.15	-	-	-	-	-	-	-	-	-	-
CoT 2025.02	72.3	89	-	82.4	-	-	-	61.6	75.3	56.2	69.3
SFT → A-GRPO 2025.09	72.06	-	-	-	-	-	-	-	-	-	-
SFT → GRPO 2025.09	71.54	-	-	-	-	-	-	-	-	-	-
GRPO 2025.09	71.43	-	-	-	-	-	-	-	-	-	-
A-SFT 2025.09	70.34	-	-	-	-	-	-	-	-	-	-
A-CHORD 2025.09	70.32	-	-	-	-	-	-	-	-	-	-
Qwen3-4B 2025.09	70.31	-	-	-	-	-	-	-	-	-	-
MapCoder 2025.02	70.2	80.5	71.3	78.3	-	-	-	70.1	74	54.4	66.4
SFT 2025.09	68.62	-	-	-	-	-	-	-	-	-	-
Self-Planning 2025.02	68.4	85.4	-	75.8	-	-	-	62.2	73.8	50.4	63.1
CoT 2025.02	67.2	70.7	68.3	78.3	-	-	-	63.4	67.5	55.7	67
MAmmoTH2-7B-Plus 2024.05	66.1	-	-	-	-	-	58.2	-	-	-	-
Direct 2025.02	65.9	71.3	67.1	75.8	-	-	-	64.6	67.7	52.6	64.2
Llama-3-8B-Instruct 2024.05	65.8	-	-	-	-	-	58	-	-	-	-
Self-Planning 2025.02	63.6	70.7	62.8	73.8	-	-	-	61	64.8	51.1	62.5
MAmmoTH2-8x7B-Plus 2024.05	63.3	-	-	-	-	-	55.3	-	-	-	-
Self-collaboration 2025.02	62.1	74.4	-	68.2	-	-	-	56.1	65.3	49.5	58.9
MAmmoTH2-8B-Plus 2024.05	61.9	-	-	-	-	-	53.3	-	-	-	-
DeRS-LM⁺ Coder-MoE 2025.03	60.9	65.9	62.2	63.4	51.9	-	-	-	-	-	-
DeRS-SM⁺ Coder-MoE 2025.03	60.3	64.6	60.4	63.7	52.4	-	-	-	-	-	-
Vanilla Coder-MoE 2025.03	60.2	64.6	61	63.9	51.4	-	-	-	-	-	-
Analogical 2025.02	59.9	67.1	59.1	69.3	-	-	-	59.1	61.8	46.9	58.1
C3A 2024.07	58.7	59.5	53.8	65.4	56.1	-	-	-	-	-	-
DoRA 2024.07	58.4	59.1	52.9	65.7	55.8	-	-	-	-	-	-
LoRA 2024.07	57.6	57.9	52.4	64.8	55.3	-	-	-	-	-	-
VeRA 2024.07	56.2	56.3	50.8	64	53.6	-	-	-	-	-	-
Analogical 2025.02	54.3	66.5	62.2	58.4	-	-	-	48.8	59.1	40.3	49.4
Mixtral-8x7B-Instruct-v0.1 2024.05	52.3	-	-	-	-	-	44.7	-	-	-	-
Tulu-2-DPO-70B 2024.05	51.2	-	-	-	-	-	43	-	-	-	-
Zero-Shot 2024.07	44	33.5	29.3	61.4	51.6	-	-	-	-	-	-
Mistral-7B-Instruct-v0.2 2024.05	43.4	-	-	-	-	-	36.5	-	-	-	-
Yi-34B-Chat 2024.05	38.7	-	-	-	-	-	32.6	-	-	-	-
A-SFT 2025.09	33.93	-	-	-	-	-	-	-	-	-	-
A-SFT → A-GRPO 2025.09	33.72	-	-	-	-	-	-	-	-	-	-
C3A 2024.07	32.8	33.8	29	36.5	31.9	-	-	-	-	-	-
DoRA 2024.07	32.7	33.2	28.8	36.8	31.8	-	-	-	-	-	-
Qwen3-0.6B 2025.09	32.63	-	-	-	-	-	-	-	-	-	-
LoRA 2024.07	31.4	31.7	28	35.4	30.4	-	-	-	-	-	-
Llama-2-70b-chat 2024.05	31.4	-	-	-	-	-	26.5	-	-	-	-
VeRA 2024.07	29.5	28.9	24.6	35.1	29.6	-	-	-	-	-	-
SFT 2025.09	29.43	-	-	-	-	-	-	-	-	-	-
SFT → A-GRPO 2025.09	29	-	-	-	-	-	-	-	-	-	-
CHORD 2025.09	26.09	-	-	-	-	-	-	-	-	-	-
A-CHORD 2025.09	23.89	-	-	-	-	-	-	-	-	-	-
SFT → GRPO 2025.09	21.6	-	-	-	-	-	-	-	-	-	-
FedMomentum 2026.03	21.34	17.07	-	25.6	-	-	-	-	-	-	-
FFA-LoRA 2026.03	20.33	15.85	-	24.8	-	-	-	-	-	-	-
FedEx-LoRA 2026.03	20.23	16.46	-	24	-	-	-	-	-	-	-
A-GRPO 2025.09	19.79	-	-	-	-	-	-	-	-	-	-
FedIT 2026.03	19.62	15.24	-	24	-	-	-	-	-	-	-
RoLoRA 2026.03	19.32	15.24	-	23.4	-	-	-	-	-	-	-
Zero-Shot 2024.07	18.8	11	9.8	30.2	24.1	-	-	-	-	-	-
FLoRA 2026.03	18.21	13.41	-	23	-	-	-	-	-	-	-
Pre-trained 2026.03	17.8	12.19	-	23.4	-	-	-	-	-	-	-
GRPO 2025.09	17.58	-	-	-	-	-	-	-	-	-	-
Chat 2025.02	-	7.9	-	16.5	-	-	-	-	-	-	-
Math 2025.02	-	15.9	-	28.8	-	-	-	-	-	-	-
Code 2025.02	-	10.4	-	21.3	-	-	-	-	-	-	-
Task Arithmetic 2025.02	-	9.1	-	24.1	-	-	-	-	-	-	-
Sens-Merging (Task Arithmetic) 2025.02	-	20.1	-	42.6	-	-	-	-	-	-	-
Ties-Merging 2025.02	-	21.3	-	40.4	-	-	-	-	-	-	-
Sens-Merging (Ties-Merging) 2025.02	-	20.6	-	41.3	-	-	-	-	-	-	-
DARE 2025.02	-	20.1	-	39.1	-	-	-	-	-	-	-
Sens-Merging (DARE) 2025.02	-	20.7	-	39.3	-	-	-	-	-	-	-
LATS 2025.02	-	83.8	-	-	-	-	-	-	-	-	-
LATS 2025.02	-	92.7	-	-	-	-	-	-	-	-	-
LATS 2025.02	-	88.8	-	-	-	-	-	81.2	85	-	-