Share your thoughts, 1 month free Claude Pro on usSee more

Audio Reasoning on MMAR (test)

74.4Average Score

Gemini 2.5 Pro

Updated 1mo ago

Evaluation Results

Method
Gemini 2.5 Pro 2026.04	74.4	67.3	56.8	82	100	84.9	80.5	66.7	-	-
Audio-DeepThinker 2026.04	74	66.06	64.08	80.27	81.82	80.73	73.17	75	-	-
Audio-Cogito 2026.04	71.7	66.67	53.4	79.25	90.91	79.9	76.83	79.17	62.22	87
Qwen3-Omni-30B-A3B-Instruct 2026.04	70.1	62.42	57.28	77.21	81.82	79.82	64.63	70.83	-	-
Gemini 2.5 Flash 2026.04	68.4	60	53.4	77.2	63.6	76.2	69.5	75	-	-
Qwen3-Omni-Thinking 2026.04	68	64.24	50	79.25	54.55	72.48	69.51	70.83	57.97	85
Qwen3-Omni-30B-A3B-Thinking 2026.04	67.8	61.21	47.09	77.89	72.73	77.98	68.29	70.83	-	-
AudioMCQ 2026.04	67.1	-	-	-	-	-	-	-	-	-
Gemini 2.5 Flash 2026.04	67	58.18	47.09	76.53	63.64	77.52	76.83	54.17	-	-
Qwen3-Omni-Instruct 2026.04	66.9	59.39	54.37	72.45	63.67	77.52	65.85	66.67	-	-
Gemini 2.0 Flash 2026.04	65.6	61.2	51	72.1	81.8	72.5	65.9	70.8	-	-
Gemini 2.0 Flash 2026.04	65.6	61.21	50.97	72.11	81.82	72.48	65.85	70.83	-	-
Audio-Thinker 2026.04	65.3	68.48	53.88	64.29	72.73	71.56	73.17	66.67	-	-
AudioRouter 2026.02	64.4	61.4	66.1	77.2	57.2	65.1	68.3	55.5	-	-
GPT-4o Audio 2026.04	63.5	53.9	51	70.4	63.6	72.5	62.2	75	-	-
GPT-4o Audio 2026.04	63.5	53.94	50.97	70.41	63.64	72.48	62.2	75	-	-
Omni-R1 2026.04	63.4	67.3	51.5	64.3	45.5	70.2	64.6	70.8	-	-
Omni-CLST 2026.02	63	58.2	64.7	76.5	54.9	63.2	67.4	56.1	-	-
AuTAgent 2026.02	63	51.52	45.15	73.81	-	-	-	-	-	-
CESAR 2026.04	62.7	66.06	55.83	62.24	63.64	67.43	60.98	66.67	-	-
Omni-R1 2026.04	61.2	63.6	51.5	62.2	72.7	65.1	62.2	70.8	-	-
Ke-Omni-R 2026.04	60.9	63.64	47.09	62.93	63.64	68.35	67.07	45.84	-	-
Tools w/ Desc. 2026.02	60	50.91	43.2	72.79	-	-	-	-	-	-
Step-Audio-R1 2026.04	58.6	32.52	32.52	68.71	45.22	72.02	62.2	72.02	46.55	79
Audio Flamingo 3 2026.04	58.5	-	-	-	-	-	-	-	-	-
Kimi-Audio 2026.04	57.6	-	-	-	-	-	-	-	-	-
Audio Flamingo 3 2026.04	57.4	56.97	45.15	59.52	45.45	67.89	59.76	41.67	45.2	79
Qwen2.5-Omni 2026.04	56.7	58.8	40.8	59.9	54.5	61.9	67.1	58.3	-	-
Qwen-2.5-Omni 2026.04	56.7	58.79	40.78	59.86	54.55	61.93	67.07	58.33	-	-
Covo-Audio 2026.04	55.3	57.58	41.26	62.5	36.36	59.17	58.54	62.5	-	-
Vanilla (No-Tool) 2026.02	55	43.64	35.44	67.01	-	-	-	-	-	-
Random Selection 2026.02	52.2	42.42	30.58	61.22	-	-	-	-	-	-
Qwen2.5-Omni 2026.02	52.1	45.6	55.2	68.4	42.1	51.5	55.8	46.1	-	-
AudioRouter 2026.02	51.8	46.5	54.6	67.1	44.1	50.9	54.3	45.4	-	-
R1-AQA 2026.02	51.3	46.2	54.1	66.8	43.5	50.2	53.7	44.6	-	-
GPT-4o mini Audio 2026.04	50.6	38.8	35.9	58.8	45.5	60.1	57.3	60	-	-
GPT-4o mini Audio 2026.04	50.6	38.79	35.92	58.84	45.45	60.09	57.32	50	-	-
All-Tools (Concat.) 2026.02	49.7	38.79	29.61	64.29	-	-	-	-	-	-
AuTAgent 2026.02	41.3	48.48	31.55	45.92	-	-	-	-	-	-
Audio-Reasoner 2026.04	40.5	42.42	32.52	42.52	45.13	48.62	30.49	29.17	28.4	68
Random Selection 2026.02	38.6	47.27	30.1	40.82	-	-	-	-	-	-
Tools w/ Desc. 2026.02	37.5	40	27.67	40	-	-	-	-	-	-
Audio-Reasoner 2026.04	36.8	43.64	33.5	32.99	45.45	42.66	31.71	25	-	-
All-Tools (Concat.) 2026.02	35.5	40	25.24	37.07	-	-	-	-	-	-
Vanilla (No-Tool) 2026.02	35.1	37.58	22.33	38.78	-	-	-	-	-	-
SALMONN 2026.04	32.8	30.9	29.6	34.4	9.1	37.6	28.1	37.5	-	-
Audio-CoT 2026.04	31.3	35.8	25.2	34	9.1	30.7	30.5	37.5	19.85	62
Qwen2-Audio 2026.02	30.4	28.5	30.2	35.4	26.8	31	32.5	28.4	-	-
Qwen2-Audio 2026.04	30.4	33.9	23.3	33	9.1	33	26.8	33.3	-	-
Qwen2-Audio-Instruct 2026.04	30	33.3	24.3	32.3	9.1	31.2	30.5	25	-	-
Mellow 2026.04	30	33.3	26.7	24.8	18.2	37.2	32.9	29.2	18.5	61
Qwen2-Audio-Instruct 2026.04	30	33.33	24.27	32.31	9.09	31.19	30.49	25	-	-
Random Guess 2026.04	29.32	29.39	25.88	31.48	25	29.3	31.1	28.13	-	-
Random Guess 2026.02	29.3	29.4	25.9	31.5	25	29.3	31.1	28.1	-	-
Audio Flamingo 2026.04	26.6	32.7	21.8	24.8	18.2	30.3	24.4	25	-	-
Audio Flamingo 2 2026.04	21.9	24.9	17.5	20.8	18.2	26.6	23.2	8.3	-	-
Audio Flamingo 2 2026.04	21.9	24.85	17.48	20.75	18.18	26.61	23.17	8.33	-	-