Share your thoughts, 1 month free Claude Pro on usSee more

Zero-shot Classification on 20-task CLIP evaluation suite (test)

76.61SUN Accuracy

WUDI-Merging w/ E-PMQ

Updated 15d ago

Evaluation Results

Method
WUDI-Merging w/ E-PMQ 2026.05	76.61	84.74	90.65	94.48	93.96	84.35	98.73	70.74	94.99	82.87	40.44	95.8	99.2	84.86	98.18	91.17	91.98	48.37	17.02	79.63	80.94
Simple Averaging w/ E-PMQ 2026.05	72.12	82.74	83.48	79.67	94.5	71.14	97.93	65.16	92.31	80.59	40.9	95.31	98.83	84.11	98.16	92.68	90.27	49.53	13.02	70.4	77.64
Task Arithmetic w/ E-PMQ 2026.05	72.06	83.15	82.75	76.63	93.5	70.18	97.8	64.57	91.97	73.44	40.47	95.31	98.71	82.76	98.01	91.91	89.91	48.57	12.44	70.02	76.71
TIES-Merging w/ E-PMQ 2026.05	71.99	82.89	84.06	80	94.51	71.1	97.9	65.59	92.36	78.2	40.47	95.39	98.73	83.88	98.19	92.56	90.35	47.65	13.13	70.07	77.45
Simple Averaging 2026.05	70.7	77.7	76.4	75.3	69.5	62.1	93.7	57.7	80.8	73.6	52.7	94.2	99.2	81.7	97	90.7	77.4	16.1	10.4	66.1	71.1
Simple Averaging w/ GPTQ 2026.05	70.32	76.59	76.46	71.33	68.28	60.59	93.14	58.03	79.9	72.97	35.51	94.66	99.16	81.31	96.88	92.18	74.46	16.16	10.55	68.81	69.86
WUDI-Merging 2026.05	70.3	72.1	73.3	69.7	81.9	84.3	98.1	56.8	85.9	83.7	64.2	94.5	97.5	72.9	95.7	83.5	89.6	33.2	33.3	74.8	75.8
Simple Averaging w/ RTN 2026.05	69.9	76.84	76.1	69.74	65.49	60.88	93.61	57.07	79.1	59.23	33.51	94.14	99.04	80.82	96.83	91.82	74.09	15.61	9.22	65.02	68.4
WUDI-Merging w/ GPTQ 2026.05	69.77	70.43	73.49	63.78	81.69	73.64	98.11	57.55	84.88	76.66	39.2	94.03	97.29	72.45	95.53	82.37	89.98	30.9	13.39	78.64	72.19
WUDI-Merging w/ RTN 2026.05	69.19	68.75	71.27	62.07	80.09	72.41	98.05	55.59	83.67	74.26	38.92	94.17	97.25	71.49	95.32	80.92	89.64	30.03	13.35	78.36	71.24
WUDI-Merging w/ AWQ 2026.05	66.49	57.02	68.78	55.52	78.56	66.7	98.19	56.12	81.64	74.29	39.36	92.61	96.78	67.36	94.22	77.79	86.65	30.93	14.12	78.09	69.06
Simple Averaging w/ AWQ 2026.05	66.07	63.61	62.3	47.15	48.4	46.14	86.97	52.71	73.28	54.83	36.15	92.31	96.98	70.23	93.34	83.94	63.21	12.17	10.84	65.07	61.29
TIES-Merging 2026.05	64.4	56.6	49.1	42.1	67.2	56.8	95.3	46.2	64.9	78.3	54.9	91.3	95.7	62.9	92.9	70.5	82	19.9	10.9	56.9	63
TIES-Merging w/ GPTQ 2026.05	63.89	55.34	48.97	40.19	66.72	51.81	95.36	47.87	65.56	50.09	38.56	90.76	95.6	62.88	92.56	70.72	81.56	17.94	9.88	75.34	61.08
TIES-Merging w/ RTN 2026.05	63.03	53.95	48.51	40.48	63.77	51.18	94.76	46.81	63.07	50.03	38.14	90.98	95.3	61.95	92	69.22	81.65	17.34	10.01	73.31	60.27
TIES-Merging w/ AWQ 2026.05	60.02	44.1	44.32	33.59	60.25	45.9	95.88	45.37	64.42	50.08	39.37	87.95	94.46	60.35	90.73	64.43	77.84	19.09	11.07	71.61	58.04
Task Arithmetic w/ GPTQ 2026.05	22.64	13.72	18.81	26.3	19.21	24.79	78.34	22.45	21.56	50.02	29.51	57.26	75.83	25.32	64.81	16.34	56.88	12.07	10.63	50.36	34.84
Task Arithmetic w/ RTN 2026.05	21.65	11.91	17.94	25	18.75	23.87	76.87	22.45	21.17	50.02	27.79	55.82	73.36	23.19	61.26	16.08	53.85	12.46	10.5	50.19	33.71
Task Arithmetic 2026.05	17.7	17.4	45.9	89.9	59.5	59.3	96.1	36.6	41.3	81.7	49.1	65.7	81.8	28	76.5	24.3	80.5	62.1	73.3	57.7	57.2
Task Arithmetic w/ AWQ 2026.05	12.23	5.43	13.08	24.3	13.29	19.83	63.04	18.88	15.76	50.02	26.33	38.81	66.45	16.15	48.13	10.49	39.91	10.78	10.21	50.14	27.66