Share your thoughts, 1 month free Claude Pro on usSee more

Zero-shot Question Answering on QA Suite (PIQA, Winogrande, HellaSwag, ARC-E, ARC-C, LAMBADA)

78.24PIQA Accuracy

QuaRot

Updated 13d ago

Evaluation Results

Method
QuaRot 2026.05	78.24	68.98	73.2	55.77	41.64	56.67	62.42
FP32 2026.05	78.07	69.14	73.3	55.85	42.49	58.08	62.82
Quant.npu 2026.05	77.64	68.67	72.78	55.01	43.09	57.5	62.45
SpinQuant 2026.05	77.37	68.59	72.94	54.21	41.72	58.24	62.18
Quant.npu 2026.05	76.33	66.85	69.2	51.01	40.61	50.53	59.06
QuaRot 2026.05	76.33	66.46	70.47	54.71	40.36	55.25	60.6
SpinQuant 2026.05	76.28	66.77	70.2	55.51	41.72	56.16	61.11
FP32 2026.05	76.22	68.35	71.27	54.17	41.81	57.03	61.48
executorch 2026.05	76.06	67.88	71.72	55.22	43.17	55.79	61.64
Quant.npu 2026.05	76.01	65.51	68.59	53.96	39.42	54.84	59.72
executorch 2026.05	75.95	64.88	69.33	53.62	40.44	51.78	59.33
FP32 2026.05	75.41	67.48	70.02	55.93	41.98	56.96	61.3
Quant.npu 2026.05	75.41	68.27	70.03	54.46	41.04	60.66	61.65
executorch 2026.05	74.97	67.25	72.03	56.19	41.81	57.07	61.55
QuaRot 2026.05	74.48	66.54	69.58	54.67	41.55	56.39	60.54
Quant.npu 2026.05	74.37	62.27	66.19	51.05	38.74	44.4	56.17
SpinQuant 2026.05	74.05	66.77	69.36	55.01	41.13	55.77	60.35
Quant.npu 2026.05	73.78	64.96	67.77	51.05	38.91	53.17	58.27
executorch 2026.05	73.72	66.38	69.4	55.18	42.66	55.27	60.44
executorch 2026.05	73.61	67.72	68.78	53.28	39.42	53.29	59.35
SpinQuant 2026.05	73.34	61.09	64.14	58.08	40.78	40.17	56.27
Quant.npu 2026.05	73.12	61.56	60.96	48.65	35.84	45.62	54.36
QuaRot 2026.05	72.2	60.46	64.5	48.15	36.26	38.27	53.31
Quant.npu 2026.05	72.09	62.19	64.2	49.16	36.95	46.89	55.25
FP32 2026.05	71.6	60.93	59.32	54.38	39.93	47.31	55.58
executorch 2026.05	70.62	61.88	62.56	51.94	39.33	38.89	54.2
executorch 2026.05	70.4	60.46	62.8	47.98	36.35	38.35	52.72
Quant.npu 2026.05	69.48	57.7	53.42	45.41	32.51	34.97	48.98
Quant.npu 2026.05	68.72	60.14	57.1	53.54	39.59	44.3	53.9
Quant.npu 2026.05	67.79	56.04	50.54	46.51	33.11	27.29	46.88
QuaRot 2026.05	67.08	56.91	57.55	46.09	32.34	26	47.66
QuaRot 2026.05	66.59	57.06	54.56	53.16	37.03	38.33	51.12
SpinQuant 2026.05	66.59	58.8	54.68	52.19	36.01	38.04	51.05
executorch 2026.05	66.54	55.96	53.84	48.44	34.73	31.96	48.58
executorch 2026.05	65.89	55.88	52.94	49.07	34.47	34.68	48.82
SpinQuant 2026.05	63.93	56.51	56.2	41.79	29.78	29.15	46.23
SpinQuant 2026.05	63.17	52.57	41.05	36.62	29.35	9.74	38.75
Quant.npu 2026.05	62.79	52.41	46.77	46.93	32.68	21.99	43.93
executorch 2026.05	62.02	50.59	44.7	41.71	30.12	4.44	38.93
executorch 2026.05	60.94	50.59	41.79	40.61	27.05	12.3	38.88
QuaRot 2026.05	59.19	50.43	38.27	35.65	25.17	5.07	35.63
SpinQuant 2026.05	58.6	50.2	36.83	34.18	26.28	7.53	35.6
QuaRot 2026.05	57.94	50.04	34.65	32.83	24.74	11.16	35.23
QuaRot 2026.05	57.83	50.59	35.46	34.09	26.02	12.5	36.08
SpinQuant 2026.05	56.37	50.91	33.64	31.73	24.57	12.09	34.89
executorch 2026.05	51.96	51.38	24.93	26.01	24.74	0	29.84
QuaRot 2026.05	51.96	50.91	27.68	26.89	25.17	0.14	30.46
SpinQuant 2026.05	51.69	53.28	27.6	27.27	25.34	1.51	31.12
QuaRot 2026.05	51.63	50.2	25.68	25.63	25.94	0.06	29.86
SpinQuant 2026.05	51.03	48.78	27.31	26.14	22.95	0.19	29.4
executorch 2026.05	50.33	49.01	25.18	24.83	22.53	0	28.65
executorch 2026.05	50.16	50.04	25.21	25	25.68	0	29.35
executorch 2026.05	50	51.14	25.37	24.24	23.46	0	29.04
SpinQuant 2026.05	49.89	48.93	24.99	25.29	22.61	0.04	28.62
executorch 2026.05	49.24	50.91	25.06	25.51	24.4	0	29.19
QuaRot 2026.05	49.13	49.25	27.17	26.81	25.34	0.99	29.78