SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents
About
Large language model (LLM) agents increasingly rely on agent harnesses that manage context, tools, and multi-turn execution, making tools a central interface for acting in realistic digital environments. As harness-connected tool ecosystems expand to hundreds or thousands of APIs, services, and task-specific skills, exhaustive tool schema injection becomes costly and imposes a closed-world assumption that limits agents to a predefined static inventory. Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods often fail to align isolated tool descriptions with the agent's true task intention, especially in long-horizon tasks where required capabilities emerge through decomposition, observations, and newly induced subgoals. We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns, and dynamically retrieves tools according to evolving task states. Using a unified corpus of 7,471 tools, we evaluate SING on three real-world tool-use benchmarks. SING improves Global Recall@5 by up to 59.8% and downstream success rate by up to 28.9% over baselines, while reducing full-corpus tool-schema exposure by 99.8%, demonstrating that intention-aware graph structure enables more accurate and context-efficient tool discovery in large-scale agentic ecosystems.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Agentic Task Success | MCP-Universe | Financial Success Score70 | 41 | |
| Tool-augmented agent execution | MCP-Bench | Task Fulfillment50.5 | 32 | |
| Tool Use | MCP-Atlas | Pass Rate49.6 | 19 | |
| Server discovery | MCPBench | Recall@599.36 | 6 | |
| Server discovery | MCP-Universe | Recall@50.9116 | 6 | |
| Server discovery | MCP-Atlas | Recall@558.97 | 6 |