Function Calling on BFCL (Complexity Metrics)

83.27Success Rate (Simple)

OpenFunctions-v2

Updated 1mo ago

Evaluation Results

Method
OpenFunctions-v2 2025.10	83.27	93	85.5	66	81.94
Hammer2.1-3B 2025.10	81.42	95	89.5	81.5	86.86
Hammer2.1-3B + ToolPRM (Ours) 2025.10	80.5	95.5	91.5	88	88.88
Qwen2.5-72B-Instruct 2025.10	80.25	97.5	93.5	92	90.81
GPT-4o-mini 2025.10	80.08	90.5	89.5	87	86.77
Hammer2.1-7B + Majority 2025.10	79.58	95	93.5	85	88.27
Hammer2.1-3B + Majority 2025.10	79.5	95	89	81	86.13
Hammer2.1-7B + ToolPRM (Ours) 2025.10	79.08	95.5	94.5	89	89.52
Hammer2.1-1.5B + ToolPRM (Ours) 2025.10	78.42	92	87.5	84.5	85.61
Hammer2.1-7B 2025.10	78.08	95	93.5	88	88.65
Hammer2.1-1.5B + Majority 2025.10	77.5	91.5	84.5	80	83.38
GPT-4o 2025.10	77.17	95	93.5	85	87.67
Hammer2.1-3B + Best of N (ORM) 2025.10	77.08	93.5	89.5	84	86.02
Mistral-Nemo-Instruct 2025.10	77	93.5	89.5	84.5	86.13
Hammer2.1-7B + Best of N (ORM) 2025.10	76.58	94.5	91.5	87	87.4
Qwen2.5-7B-Instruct 2025.10	75.33	94.5	91.5	84.5	86.46
Hammer2.1-1.5B + Best of N (ORM) 2025.10	75.25	90	86.5	84	83.94
Hammer2.1-1.5B 2025.10	74.67	92	84.5	80	82.79
Hammer2.1-7B + Token-level Beam Search 2025.10	74.58	93.5	91.5	82.5	85.52
Hammer2.1-3B + Token-level Beam Search 2025.10	74.5	91	87.5	77	82.5
Qwen2.5-3B-Instruct 2025.10	74.17	90.5	79.5	79	80.79
xLAM-7b-fc-r 2025.10	73.08	93.5	87	84	84.4
Llama-3.1-8B-Instruct 2025.10	72.83	93.5	87	83.5	84.21
Qwen2.5-32B-Instruct 2025.10	72.83	94	93.5	88.5	87.21
GRANITE 2025.10	72.83	91.5	84	81.5	82.46
Qwen2.5-1.5B-Instruct 2025.10	72.42	87	81.5	75.5	79.11
Hammer2.1-1.5B + Token-level Beam Search 2025.10	72.33	91.5	81	73.5	79.58
xLAM-1b-fc-r 2025.10	69.67	89.5	79	66.5	76.17
Hammer2.1-0.5B 2025.10	68	83	71.5	54	69.13