Share your thoughts, 1 month free Claude Pro on usSee more

Code Completion on APC Placeholder Completion 1.0 (test)

68.26PCR

Qwen2.5-Coder-7B-GRPO

Updated 2mo ago

Evaluation Results

Method
Qwen2.5-Coder-7B-GRPO 2026.04	68.26	47.06	32.12	78.72	38.18	36.38
StarCoder2-15B-GRPO 2026.04	67.88	45.35	30.78	77.49	36.67	35.82
StarCoder2-15B-SFT 2026.04	65.71	44.93	29.52	75.08	35.63	38.3
Qwen2.5-Coder-14B-GRPO 2026.04	65.2	50.15	32.7	79.05	39.58	34.54
CodeLlama-13B-GRPO 2026.04	64.05	36.12	23.14	75.53	28.21	36.68
Qwen2.5-Coder-7B-SFT 2026.04	63.1	45.76	28.87	76.24	35.4	37.73
CodeLlama-7B-GRPO 2026.04	61.57	34.16	21.03	74.13	26.04	37.59
Qwen2.5-Coder-1.5B-GRPO 2026.04	60.23	45.71	27.53	74.95	34.37	36.75
StarCoder2-3B-GRPO 2026.04	59.05	37.9	22.38	69.26	28.14	37.57
CodeLlama-7B-SFT 2026.04	58.32	32.46	18.93	71	23.91	38.46
CodeLlama-13B-SFT 2026.04	58.32	31.48	18.36	71.63	23.19	37.29
Qwen2.5-Coder-14B-SFT 2026.04	53.35	48.39	25.81	76.53	33.67	35.15
CodeGemma-2B-GRPO 2026.04	51.63	37.41	19.31	72.11	25.47	37.06
StarCoder2-3B-SFT 2026.04	49.52	33.65	16.67	66.22	22.29	39.02
StarCoder2-7B-GRPO 2026.04	49.33	11.47	20.46	75.03	27.4	35.4
CodeGemma-7B-GRPO 2026.04	47.61	42.57	20.27	74.57	27.46	34.71
Qwen2.5-Coder-3B-GRPO 2026.04	43.59	50.88	22.18	76.56	30.89	36.6
CodeGemma-7B-SFT 2026.04	38.24	31	11.85	65.13	17.15	38.14
StarCoder2-7B-SFT 2026.04	38.1	33.75	12.86	68.36	18.62	37.31
Qwen2.5-Coder-1.5B-SFT 2026.04	35.37	36.76	13	70.32	19.21	38.08
CodeGemma-2B-SFT 2026.04	13.38	30	4.02	53.44	7.08	44.06
Qwen2.5-Coder-3B-SFT 2026.04	12.81	41.79	5.35	67.84	9.49	40.61
Qwen2.5-Coder-3B-BASE 2026.04	2.49	0	0	54.16	0	45.42
Qwen2.5-Coder-1.5B-BASE 2026.04	0.19	0	0	21.76	0	72.93
Qwen2.5-Coder-7B-BASE 2026.04	0.19	0	0	35.1	0	60.75
Qwen2.5-Coder-14B-BASE 2026.04	0	0	0	31.72	0	62.82
CodeGemma-2B-BASE 2026.04	0	0	0	44.25	0	55.85
CodeGemma-7B-BASE 2026.04	0	0	0	44.72	0	48.92
CodeLlama-7B-BASE 2026.04	0	0	0	24.75	0	68.14
CodeLlama-13B-BASE 2026.04	0	0	0	26.98	0	65.99
StarCoder2-3B-BASE 2026.04	0	0	0	26.62	0	67.36
StarCoder2-7B-BASE 2026.04	0	0	0	30.75	0	62.81
StarCoder2-15B-BASE 2026.04	0	0	0	22.4	0	69.89