Share your thoughts, 1 month free Claude Pro on usSee more

Code Generation on HumanEval+ (test)

98.1Pass@1

SKETCHVERIFY (B: sem. vote)

Updated 5d ago

Evaluation Results

Method
SKETCHVERIFY (B: sem. vote) 2026.05	98.1	-	-	-	-	-	-	-	-
Greedy 2026.05	97.4	-	-	-	-	-	-	-	-
Majority Vote 2026.05	97.4	-	-	-	-	-	-	-	-
Semantic Voting (flat) 2026.05	97.4	-	-	-	-	-	-	-	-
Majority Vote 2026.05	97	-	-	-	-	-	-	-	-
Semantic Voting (flat) 2026.05	97	-	-	-	-	-	-	-	-
Greedy 2026.05	96.3	-	-	-	-	-	-	-	-
SKETCHVERIFY (B: sem. vote) 2026.05	96.3	-	-	-	-	-	-	-	-
Best-of-N 2026.05	94.8	-	-	-	-	-	-	-	-
Best-of-N 2026.05	93.9	-	-	-	-	-	-	-	-
Majority Vote 2026.05	92.7	-	-	-	-	-	-	-	-
Semantic Voting (flat) 2026.05	92.7	-	-	-	-	-	-	-	-
SKETCHVERIFY (B: sem. vote) 2026.05	92.1	-	-	-	-	-	-	-	-
Greedy 2026.05	85.4	-	-	-	-	-	-	-	-
ACES-O 2026.04	84.76	-	-	-	-	84.76	86.59	-	-
Majority Voting 2026.04	84.15	-	-	-	-	84.76	87.2	-	-
ACES-C 2026.04	84.15	-	-	-	-	84.15	86.59	-	-
GPT4-Turbo 2023.12	81.7	-	-	-	-	-	-	-	-
Best-of-N 2026.05	81.7	-	-	-	-	-	-	-	-
ACES-O 2026.04	81.1	-	-	-	-	82.93	83.54	-	-
ACES-C 2026.04	79.88	-	-	-	-	82.32	82.93	-	-
Zero-shot 2026.04	79.4	-	-	-	-	82.97	85.96	-	-
Majority Voting 2026.04	78.66	-	-	-	-	82.32	82.93	-	-
ACES-C 2026.04	78.66	-	-	-	-	78.66	79.27	-	-
ACES-O 2026.04	76.22	-	-	-	-	76.22	78.66	-	-
Zero-shot 2026.04	75.63	-	-	-	-	81.38	85.62	-	-
MPSC-Lexical 2023.09	74.39	-	-	-	-	75	77.24	-	-
Majority Voting 2026.04	74.39	-	-	-	-	75.61	78.05	-	-
MPSC-Semantic 2023.09	73.54	-	-	-	-	74.46	75.26	-	-
MPSC-Label 2023.09	73.47	-	-	-	-	76.66	77.25	-	-
Teacher 2026.05	71.34	-	-	-	-	-	-	-	-
IF-CL 2023.12	71.3	-	-	-	-	-	-	-	-
Zero-shot 2026.04	71.05	-	-	-	-	75.3	79.03	-	-
MT (DS-Coder-6.9B-Inst) 2026.04	70.7	-	-	-	-	-	-	-	-
GPT-4 2023.09	70.52	-	-	-	-	75.48	79.54	-	-
IF-CLP 2023.12	69.5	-	-	-	-	-	-	-	-
IF-CLP 2023.12	69.5	-	-	-	-	-	-	-	-
WeDLM-8B 2025.12	68.9	-	-	-	-	-	-	-	-
IF-CL 2023.12	68.3	-	-	-	-	-	-	-	-
CodeT 2023.09	67.87	-	-	-	-	68.75	69.65	-	-
MagicoderS-CLP 2023.12	66.5	-	-	-	-	-	-	-	-
GPT3.5-Turbo 2023.12	65.9	-	-	-	-	-	-	-	-
MPSC-Uniform 2023.09	65.05	-	-	-	-	69.76	71.72	-	-
WizardCoder-CLP 2023.12	64.6	-	-	-	-	-	-	-	-
WeDLM-7B 2025.12	64	-	-	-	-	-	-	-	-
Self-consistency 2023.09	63.5	-	-	-	-	64.7	65.67	-	-
Qwen3-8B 2025.12	63.4	-	-	-	-	-	-	-	-
MBR-EXEC 2023.09	62.12	-	-	-	-	67.08	71.38	-	-
GPT-3.5-Turbo 2023.09	58.75	-	-	-	-	66.58	73.96	-	-
Qwen2.5-7B 2025.12	53.05	-	-	-	-	-	-	-	-
LLaDA-1.5 2025.12	45.1	-	-	-	-	-	-	-	-
CODELLAMA-PYTHON 2023.12	42.7	-	-	-	-	-	-	-	-
ESPO 2025.12	42.7	-	-	-	-	-	-	-	-
LLaDA 2025.12	41.5	-	-	-	-	-	-	-	-
WizardCoder-CLP 2023.12	40.9	-	-	-	-	-	-	-	-
VCRD-Prob 2026.04	40.9	-	-	-	-	-	-	-	-
α-β divergence + Ours 2026.05	38.41	-	-	-	-	-	-	-	-
DistilLLM 2026.04	38.4	-	-	-	-	-	-	-	-
DistillLM-2 2026.04	38.4	-	-	-	-	-	-	-	-
ABKD 2026.04	37.8	-	-	-	-	-	-	-	-
diffu-GRPO(d1) 2025.12	37.2	-	-	-	-	-	-	-	-
KD 2026.04	37.2	-	-	-	-	-	-	-	-
CODELLAMA-PYTHON 2023.12	36.6	-	-	-	-	-	-	-	-
ESPO 2025.12	36.6	-	-	-	-	-	-	-	-
Forward KL + Ours 2026.05	36.59	-	-	-	-	-	-	-	-
α-β divergence 2026.05	36.59	-	-	-	-	-	-	-	-
Total Variation + Ours 2026.05	36.16	-	-	-	-	-	-	-	-
SeqKD 2026.04	36	-	-	-	-	-	-	-	-
GKD 2026.04	36	-	-	-	-	-	-	-	-
Skew FKL 2026.05	35.98	-	-	-	-	-	-	-	-
Skew RKL 2026.05	35.98	-	-	-	-	-	-	-	-
Skew RKL + Ours 2026.05	35.98	-	-	-	-	-	-	-	-
Skew FKL + Ours 2026.05	35.75	-	-	-	-	-	-	-	-
Total Variation 2026.05	35.37	-	-	-	-	-	-	-	-
Reverse KL 2026.05	34.15	-	-	-	-	-	-	-	-
Reverse KL + Ours 2026.05	34.15	-	-	-	-	-	-	-	-
CODELLAMA-PYTHON 2023.12	34.1	-	-	-	-	-	-	-	-
Forward KL 2026.05	33.54	-	-	-	-	-	-	-	-
Hard KD 2026.05	32.93	-	-	-	-	-	-	-	-
wd1 2025.12	32.9	-	-	-	-	-	-	-	-
LLaDA-1.5 2025.12	32.3	-	-	-	-	-	-	-	-
Proxy (Tokenizer) 2026.02	30.5	-	-	-	-	-	-	-	-
LLaDA 2025.12	30.5	-	-	-	-	-	-	-	-
Proxy (Neural) 2026.02	29.9	-	-	-	-	-	-	-	-
diffu-GRPO(d1) 2025.12	29.9	-	-	-	-	-	-	-	-
wd1 2025.12	29.9	-	-	-	-	-	-	-	-
wd1 2025.12	29.9	-	-	-	-	-	-	-	-
StarCoder 2023.12	29.3	-	-	-	-	-	-	-	-
Tokenizer-based 2026.02	29.3	-	-	-	-	-	-	-	-
MS (DS-Coder-1.3B) 2026.04	29.3	-	-	-	-	-	-	-	-
Tokenizer-based 2026.02	28.7	-	-	-	-	-	-	-	-
LLaDA-8B 2025.12	28.05	-	-	-	-	-	-	-	-
Tokenizer-based 2026.02	28	-	-	-	-	-	-	-	-
Student (No Distill) 2026.05	27.44	-	-	-	-	-	-	-	-
CodeGen-Mono 2023.12	27.4	-	-	-	-	-	-	-	-
Proxy (Neural) 2026.02	26.8	-	-	-	-	-	-	-	-
CodeT5+ 2023.12	26.2	-	-	-	-	-	-	-	-
Proxy (Tokenizer) 2026.02	26.2	-	-	-	-	-	-	-	-
Byte-level 2026.02	24.4	-	-	-	-	-	-	-	-
Proxy (Tokenizer) 2026.02	24.4	-	-	-	-	-	-	-	-

Showing 100 of 195 rows