Share your thoughts, 1 month free Claude Pro on usSee more

Code Generation on HumanEval and MBPP (Aggregated)

95.1HumanEval Score

CodeSim

Updated 1mo ago

Evaluation Results

Method
CodeSim 2025.02	95.1	87.2	90.7	-	82.7	-	-	86	89.4	61.2	76
CodeSim 2025.02	94.5	84.8	89.7	-	81.3	-	-	81.7	87	61.5	75.6
MapCoder 2025.02	93.9	83.5	83.1	-	78.6	-	-	82.9	86.8	57.7	70.4
LATS 2025.02	92.7	-	-	-	-	-	-	-	-	-	-
Reflexion 2025.02	91	81.7	78.3	-	74.4	-	-	78.7	83.8	51.9	65.1
CoT 2025.02	90.9	87.2	82.9	-	78.6	-	-	82.3	86.8	57.9	70.4
Direct 2025.02	90.2	82.3	81.1	-	76.5	-	-	81.1	84.5	55.9	68.5
MapCoder 2025.02	90.2	81.7	88.7	-	79	-	-	80.5	84.1	59.2	74
CoT 2025.02	89	-	82.4	-	72.3	-	-	61.6	75.3	56.2	69.3
Self-Planning 2025.02	89	84.1	82.6	-	77	-	-	80.5	84.5	56.4	69.5
LATS 2025.02	88.8	-	-	-	-	-	-	81.2	85	-	-
Analogical 2025.02	88.4	83.5	75.1	-	73.6	-	-	80.5	84.1	50.9	63
Reflexion 2025.02	87.2	81.1	81.1	-	76	-	-	81.1	83.1	56.7	68.9
CodeSim 2025.02	86	73.2	86.4	-	75.1	-	-	72	77.1	59.7	73.1
Self-Planning 2025.02	85.4	-	75.8	-	68.4	-	-	62.2	73.8	50.4	63.1
LATS 2025.02	83.8	-	-	-	-	-	-	-	-	-	-
DataFlow-SFT-15K(filtered) 2025.12	82.9	-	74.9	-	78.9	-	-	-	-	-	-
MapCoder 2025.02	80.5	71.3	78.3	-	70.2	-	-	70.1	74	54.4	66.4
Direct 2025.02	80.1	81.7	81.1	-	73.2	-	-	73.8	78.5	54.7	67.9
DataFlow-SFT-15K(random) 2025.12	79.9	-	75.9	-	77.9	-	-	-	-	-	-
WizardLM(filtered) 2025.12	77.4	-	80.4	-	78.9	-	-	-	-	-	-
WizardLM(random) 2025.12	75.6	-	82	-	78.8	-	-	-	-	-	-
Self-collaboration 2025.02	74.4	-	68.2	-	62.1	-	-	56.1	65.3	49.5	58.9
Alpaca(filtered) 2025.12	73.8	-	75.7	-	74.8	-	-	-	-	-	-
Alpaca(random) 2025.12	71.3	-	75.9	-	73.6	-	-	-	-	-	-
Direct 2025.02	71.3	67.1	75.8	-	65.9	-	-	64.6	67.7	52.6	64.2
CoT 2025.02	70.7	68.3	78.3	-	67.2	-	-	63.4	67.5	55.7	67
Self-Planning 2025.02	70.7	62.8	73.8	-	63.6	-	-	61	64.8	51.1	62.5
Analogical 2025.02	67.1	59.1	69.3	-	59.9	-	-	59.1	61.8	46.9	58.1
Analogical 2025.02	66.5	62.2	58.4	-	54.3	-	-	48.8	59.1	40.3	49.4
DeRS-LM⁺ Coder-MoE 2025.03	65.9	62.2	63.4	51.9	60.9	-	-	-	-	-	-
Vanilla Coder-MoE 2025.03	64.6	61	63.9	51.4	60.2	-	-	-	-	-	-
DeRS-SM⁺ Coder-MoE 2025.03	64.6	60.4	63.7	52.4	60.3	-	-	-	-	-	-
C3A 2024.07	59.5	53.8	65.4	56.1	58.7	-	-	-	-	-	-
DoRA 2024.07	59.1	52.9	65.7	55.8	58.4	-	-	-	-	-	-
LoRA 2024.07	57.9	52.4	64.8	55.3	57.6	-	-	-	-	-	-
VeRA 2024.07	56.3	50.8	64	53.6	56.2	-	-	-	-	-	-
C3A 2024.07	33.8	29	36.5	31.9	32.8	-	-	-	-	-	-
Zero-Shot 2024.07	33.5	29.3	61.4	51.6	44	-	-	-	-	-	-
DoRA 2024.07	33.2	28.8	36.8	31.8	32.7	-	-	-	-	-	-
LoRA 2024.07	31.7	28	35.4	30.4	31.4	-	-	-	-	-	-
VeRA 2024.07	28.9	24.6	35.1	29.6	29.5	-	-	-	-	-	-
Ties-Merging 2025.02	21.3	-	40.4	-	-	-	-	-	-	-	-
Sens-Merging (DARE) 2025.02	20.7	-	39.3	-	-	-	-	-	-	-	-
Sens-Merging (Ties-Merging) 2025.02	20.6	-	41.3	-	-	-	-	-	-	-	-
Sens-Merging (Task Arithmetic) 2025.02	20.1	-	42.6	-	-	-	-	-	-	-	-
DARE 2025.02	20.1	-	39.1	-	-	-	-	-	-	-	-
FedMomentum 2026.03	17.07	-	25.6	-	21.34	-	-	-	-	-	-
FedEx-LoRA 2026.03	16.46	-	24	-	20.23	-	-	-	-	-	-
Math 2025.02	15.9	-	28.8	-	-	-	-	-	-	-	-
FFA-LoRA 2026.03	15.85	-	24.8	-	20.33	-	-	-	-	-	-
FedIT 2026.03	15.24	-	24	-	19.62	-	-	-	-	-	-
RoLoRA 2026.03	15.24	-	23.4	-	19.32	-	-	-	-	-	-
FLoRA 2026.03	13.41	-	23	-	18.21	-	-	-	-	-	-
Pre-trained 2026.03	12.19	-	23.4	-	17.8	-	-	-	-	-	-
Zero-Shot 2024.07	11	9.8	30.2	24.1	18.8	-	-	-	-	-	-
Code 2025.02	10.4	-	21.3	-	-	-	-	-	-	-	-
Task Arithmetic 2025.02	9.1	-	24.1	-	-	-	-	-	-	-	-
Chat 2025.02	7.9	-	16.5	-	-	-	-	-	-	-	-
GPT-4-1106-preview 2024.05	-	-	-	-	85.6	-	77.5	-	-	-	-
GPT-3.5-Turbo-1106 2024.05	-	-	-	-	79.7	-	70.2	-	-	-	-
Tulu-2-DPO-70B 2024.05	-	-	-	-	51.2	-	43	-	-	-	-
Llama-2-70b-chat 2024.05	-	-	-	-	31.4	-	26.5	-	-	-	-
Yi-34B-Chat 2024.05	-	-	-	-	38.7	-	32.6	-	-	-	-
Mistral-7B-Instruct-v0.2 2024.05	-	-	-	-	43.4	-	36.5	-	-	-	-
Llama-3-8B-Instruct 2024.05	-	-	-	-	65.8	-	58	-	-	-	-
Mixtral-8x7B-Instruct-v0.1 2024.05	-	-	-	-	52.3	-	44.7	-	-	-	-
MAmmoTH2-7B-Plus 2024.05	-	-	-	-	66.1	-	58.2	-	-	-	-
MAmmoTH2-8B-Plus 2024.05	-	-	-	-	61.9	-	53.3	-	-	-	-
MAmmoTH2-8x7B-Plus 2024.05	-	-	-	-	63.3	-	55.3	-	-	-	-