Share your thoughts, 1 month free Claude Pro on usSee more

Zero-shot Evaluation on MobileLLM Evaluation Suite

69.19ARC-e

EDGERAZOR

Updated 27d ago

Evaluation Results

Method
EDGERAZOR 2026.04	69.19	36.26	51.91	62.26	70.4	56.2	40.74	37.4	37.96	57.41	25	53	41.94
EDGERAZOR 2026.04	69.11	35.84	51.82	62.6	70.35	56.2	40.58	37.4	37.9	57.21	24.66	45	41.86
EDGERAZOR 2026.04	65.99	32.68	45.99	62.11	68.55	56.51	40.07	35.2	39.05	56.51	24.41	99	40.62
EDGERAZOR 2026.04	65.87	32.68	45.98	61.71	68.82	56.27	40.02	35	38.97	56.53	24.27	76	40.53
MobileLLM-350M 2026.04	64.94	35.49	52.87	58.96	70.84	56.35	40.79	40.2	37.44	53.98	23.52	0	41.18
ParetoQ 2026.04	64.23	38.14	53.13	58.32	71.55	56.2	40.33	38	37.04	50.73	24.78	8	40.96
EfficientQAT 2026.04	63.68	35.67	51.73	58.47	70.73	56.75	40.74	38.4	37.11	54.07	24.16	0	40.89
ParetoQ 2026.04	62.75	33.28	51.24	60.92	70.95	56.75	39.82	39	37	46.39	25.02	0	40.24
FlatQuant 2026.04	61.95	34.73	51.72	58.93	70.84	53.75	40.33	39	38.01	52.24	23.56	15	40.4
EfficientQAT 2026.04	61.53	33.11	49.45	60.89	69.04	53.75	39.66	36.8	37.66	45.35	22.24	0	39.27
EDGERAZOR 2026.04	61.36	29.18	40.86	58.23	66.92	55.49	39.56	33.2	40.95	56.13	24.97	38	39.02
EDGERAZOR 2026.04	61.2	28.75	40.76	58.23	66.59	55.01	39.51	33	40.98	56.22	25.03	53	38.91
FlatQuant 2026.04	61.07	31.91	48.43	55.44	68.39	53.12	39.51	35.2	39.24	45.23	24.74	8	38.64
EDGERAZOR 2026.04	58.67	26.19	38.92	58.04	65.23	53.83	39.25	32	42.03	56.33	24.19	83	38.12
EDGERAZOR 2026.04	58.63	26.19	38.95	58.07	65.29	53.04	39.3	32.2	41.97	56.26	24.12	53	38.04
ParetoQ 2026.04	57.66	32.59	46.95	63.03	69.31	56.67	40.43	35.2	36.25	43.4	24.88	45	38.99
ParetoQ 2026.04	56.1	29.95	43.68	61.62	67.3	54.3	39.3	36.4	38.82	43.34	23.05	15	38
EfficientQAT 2026.04	49.92	27.05	39.29	61.77	63.49	50.91	37.56	29.6	42.5	46.05	22.93	0	36.24
FlatQuant 2026.04	31.44	22.27	27.19	44.71	51.74	47.59	34.6	25.4	49.22	43.53	23.03	0	30.82
AutoRound 2026.04	26.39	29.69	25.79	61.71	50.33	48.78	33.62	27.6	48.14	43.6	26.9	0	32.5
QTIP 2026.04	26.35	28.24	26.4	37.83	49.08	50.59	35.16	24.6	49.21	56.67	22.97	0	31.32
AutoRound 2026.04	25.88	29.27	25.86	60	49.95	51.62	33.16	26.2	47.95	46.53	27.12	0	32.58
AutoRound 2026.04	25.72	29.27	25.81	61.9	50.33	49.8	33.78	27	47.78	43.59	27	0	32.46