Share your thoughts, 1 month free Claude Pro on usSee more

Tool-augmented agent execution on MCP-Bench

53.5Task Fulfillment

Ground Truth Server

Updated 1mo ago

Evaluation Results

Method
Ground Truth Server 2026.06	53.5	72.6	74.6	74.4	62.3	55.4	-	-	-	-	-	-	73.8
SING 2026.06	50.5	67.1	67.4	69	60.8	35.3	-	-	-	-	-	-	67.5
SING 2026.06	50.1	66.8	71.4	68.3	60.1	33.4	-	-	-	-	-	-	68.4
MCP-Zero 2026.06	49.5	62.9	68.9	68.3	58.7	34.9	-	-	-	-	-	-	67.6
MCP-Zero 2026.06	48.2	61.1	68	68	57.7	33	-	-	-	-	-	-	66.7
ReAct 2026.05	46.8	36.6	41.5	31.2	40.5	19.8	77.7	0	41.7	36.4	30.1	38.9	36.8
ReAct-Plan-Exec 2026.05	38.5	38.7	42.5	47.8	36.9	20.6	100	75.4	38.6	45.2	28.8	87.7	50
SPIRAL-Exec 2026.05	34.5	47.5	55.3	58.4	33.3	27.5	100	79.9	41	56.9	30.4	89.9	54.6
ReAct-Plan-Exec 2026.05	33.7	42.6	45.2	40.4	29.6	26.8	100	85.8	38.2	42.8	28.2	92.9	50.5
SPIRAL-Exec 2026.05	32.8	51	59.6	61	33.6	30.9	100	82.8	41.9	60.3	32.2	91.4	56.5
SPIRAL-Exec 2026.05	31.6	26.6	33.6	48.4	20.5	20.2	100	100	29.1	41	20.3	100	47.6
ReAct-Plan-Exec 2026.05	31.4	47.2	53.1	65.9	27.1	29.9	100	78.4	39.3	59.5	28.5	89.2	54.1
Qwen2.5-4B 2026.06	2.4	3.58	3.42	3.78	2.48	1.45	-	-	2.99	3.6	1.97	-	2.85
Qwen2.5-4B 2026.06	1.97	2.32	3.17	3.75	1.43	1.38	-	-	2.14	3.46	1.41	-	2.34
SmolLM2-3B 2026.06	1.48	2.18	2.44	2.51	1.41	1.07	-	-	1.83	2.48	1.24	-	1.85
Llama-3.2-3B 2026.06	1.32	2.43	2.62	2.85	1.17	1.1	-	-	1.88	2.73	1.13	-	1.91
Llama-3.2-3B 2026.06	1.25	1.65	1.82	1.82	1.45	0.95	-	-	1.45	1.82	1.2	-	1.49
DeepSeek-R1-Distill-Qwen-1.5B 2026.06	1.05	1.33	1.02	1.16	1	1	-	-	1.19	1.09	1	-	1
Qwen2.5-4B 2026.06	0.946	1.53	2.01	2.95	1.05	0.929	-	-	1.24	2.48	0.987	-	1.57
Qwen2.5-4B 2026.06	0.933	1.37	2.33	2.48	0.9	0.817	-	-	1.15	2.41	0.858	-	1.47
SmolLM2-3B 2026.06	0.817	1.05	0.983	1.13	0.767	0.75	-	-	0.933	1.06	0.758	-	0.917
Llama-3.2-3B 2026.06	0.767	1.45	1.2	0.8	0.8	0.717	-	-	1.11	1	0.758	-	0.956
Qwen2.5-4B 2026.06	0.737	1.35	1.09	1.6	0.671	0.554	-	-	1.05	1.35	0.612	-	1
DeepSeek-R1-Distill-Qwen-1.5B 2026.06	0.727	0.855	0.709	0.891	0.673	0.673	-	-	0.791	0.8	0.673	-	0.692
Llama-3.2-3B 2026.06	0.717	1.02	1.28	1.23	0.733	0.75	-	-	0.867	1.26	0.742	-	0.956
SmolLM2-3B 2026.06	0.675	0.771	0.938	1.7	0.638	0.638	-	-	0.723	1.32	0.638	-	0.892
SmolLM2-3B 2026.06	0.65	0.867	0.617	0.633	0.583	0.567	-	-	0.758	0.625	0.575	-	0.653
DeepSeek-R1-Distill-Qwen-1.5B 2026.06	0.6	0.733	0.567	0.567	0.567	0.567	-	-	0.667	0.567	0.567	-	0.6
SmolLM2-3B 2026.06	0.567	0.55	0.55	0.55	0.55	0.55	-	-	0.558	0.55	0.55	-	0.553
Llama-3.2-3B 2026.06	0.55	0.85	0.917	0.867	0.583	0.583	-	-	0.7	0.892	0.583	-	0.725
DeepSeek-R1-Distill-Qwen-1.5B 2026.06	0.458	1.28	0.625	0.608	0.458	0.458	-	-	0.871	0.617	0.458	-	0.649
DeepSeek-R1-Distill-Qwen-1.5B 2026.06	0.433	0.6	0.45	0.433	0.433	0.433	-	-	0.517	0.442	0.433	-	0.464