Share your thoughts, 1 month free Claude Pro on usSee more

Code-Specific Instruction Tuning Evaluation on Magicoder Evaluation Suite

54.27ARC-C Accuracy

Pre-trained

Updated 26d ago

Evaluation Results

Method
Pre-trained 2026.05	54.27	80.09	72.62	86.2	73.3	60.98	67.14
Random 2026.05	54.1	79.64	71.55	85.47	72.69	65.2	68.95
DualSFT 2026.05	54.01	79.64	70.33	84.61	72.15	73.75	72.95
DualSFT-Data 2026.05	53.66	80.03	70.75	85.04	72.37	74.21	73.29
DualSFT-Param 2026.05	53.52	78.72	70.26	84.22	71.68	75.13	73.41
NanoAdam 2026.05	52.87	79.03	70.22	84.22	71.59	73.28	72.43
S2FT 2026.05	51.74	78.96	70.35	82.05	70.78	72.51	71.64
LoRA 2026.05	51.51	76.62	69.47	81.94	69.89	70.29	70.09
Pre-trained 2026.05	51.45	79.16	59.61	37	56.81	35.37	46.09
Random 2026.05	51.25	78.89	56.91	34.65	55.43	49.24	52.33
LIFT 2026.05	51.23	77.06	68.75	82.14	69.8	71.74	70.77
DualSFT 2026.05	51.14	79.01	59.16	35.62	56.23	56.76	56.5
DualSFT-Param 2026.05	51.13	78.54	58.14	36.34	56.04	57.52	56.78
DualSFT-Data 2026.05	51.01	78.52	58.33	36.55	56.1	56.94	56.52
LESS + S2FT 2026.05	51.01	78.66	55.68	31.14	54.12	53.12	53.62
LESS + S2FT 2026.05	50.98	79.54	69.14	81.28	70.24	67.86	69.05
FisherSFT 2026.05	50.94	76.16	68.49	81.36	69.24	71.61	70.42
LESS 2026.05	50.92	78.92	54.14	31.58	53.89	57.09	55.49
SPICE 2026.05	50.85	76.16	69.22	83.01	69.81	72.12	70.97
NanoAdam 2026.05	50.78	78.2	58.33	33.12	55.11	56.38	55.74
SPICE + NanoAdam 2026.05	50.78	78.54	57.29	33.89	55.13	52.65	53.89
SPICE + NanoAdam 2026.05	50.55	79.35	70.09	80.74	70.18	69.24	69.71
SPICE 2026.05	50.47	78.16	55.12	32.63	54.1	56.82	55.46
SMT 2026.05	50.33	76.51	68.41	82.64	69.47	73.68	71.58
LESS 2026.05	50.27	79.02	69.65	81.28	70.06	72.77	71.41
FisherSFT 2026.05	50.25	74.51	53.71	32.17	52.66	53.47	53.07
LIFT 2026.05	50.02	77.16	54.56	30.61	53.09	55.61	54.35
SMT 2026.05	49.92	77.98	56.44	32.74	54.27	56.44	55.36
S2FT 2026.05	49.74	78.29	55.75	31.09	53.72	55.29	54.5
Standard SFT 2026.05	49.22	74.31	66.14	78.64	67.08	75.56	71.32
LoRA 2026.05	49.15	75.57	56.71	31.63	53.27	54.66	53.96
Standard SFT 2026.05	47.17	76.39	53.87	24.03	50.37	58.1	54.23
Pre-trained 2026.05	43	76.61	56.52	27.45	50.9	28.66	39.78
Random 2026.05	42.99	76.09	55.31	26.99	50.35	38.59	44.47
DualSFT 2026.05	42.94	76.23	56.09	26.88	50.54	43.67	47.1
NanoAdam 2026.05	42.66	74.68	52.71	24.39	48.61	43.54	46.08
DualSFT-Data 2026.05	42.66	76.44	55.28	26.54	50.23	44.05	47.14
LoRA 2026.05	42.61	74.43	51.87	23.29	48.05	41.46	44.76
LESS + S2FT 2026.05	42.57	76.68	53.55	22.86	48.92	38.85	43.88
DualSFT-Param 2026.05	42.54	76.19	54.01	26.27	49.75	44.52	47.14
LESS 2026.05	42.18	75.66	53.97	23.46	48.82	42.66	45.74
SPICE + NanoAdam 2026.05	42.16	76.61	54.87	25.93	49.89	39.52	44.71
S2FT 2026.05	41.8	76.26	52.1	22.58	48.19	42.78	45.48
FisherSFT 2026.05	41.55	74.17	52.83	22.92	47.87	41.37	44.62
LIFT 2026.05	41.22	72.5	52.14	23.23	47.27	42.21	44.74
SMT 2026.05	40.97	74.15	52.59	23.89	47.9	43.09	45.5
SPICE 2026.05	40.54	76.09	53.13	24.52	48.57	42.32	45.45
Standard SFT 2026.05	39.99	72.09	48.18	20.69	45.24	44.68	44.96