Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks

About

Modern AI agents, driven by advances in large foundation models, promise to enhance our productivity and transform our lives by augmenting our knowledge and capabilities. To achieve this vision, AI agents must effectively plan, perform multi-step reasoning and actions, respond to novel observations, and recover from errors, to successfully complete complex tasks across a wide range of scenarios. In this work, we introduce Magentic-One, a high-performing open-source agentic system for solving such tasks. Magentic-One uses a multi-agent architecture where a lead agent, the Orchestrator, plans, tracks progress, and re-plans to recover from errors. Throughout task execution, the Orchestrator directs other specialized agents to perform tasks as needed, such as operating a web browser, navigating local files, or writing and executing Python code. We show that Magentic-One achieves statistically competitive performance to the state-of-the-art on three diverse and challenging agentic benchmarks: GAIA, AssistantBench, and WebArena. Magentic-One achieves these results without modification to core agent capabilities or to how they collaborate, demonstrating progress towards generalist agentic systems. Moreover, Magentic-One's modular design allows agents to be added or removed from the team without additional prompt tuning or training, easing development and making it extensible to future scenarios. We provide an open-source implementation of Magentic-One, and we include AutoGenBench, a standalone tool for agentic evaluation. AutoGenBench provides built-in controls for repetition and isolation to run agentic benchmarks in a rigorous and contained manner -- which is important when agents' actions have side-effects. Magentic-One, AutoGenBench and detailed empirical performance evaluations of Magentic-One, including ablations and error analysis are available at https://aka.ms/magentic-one

Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang (Eric) Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, Saleema Amershi• 2024

Related benchmarks

TaskDatasetResultRank
General AI Assistant TaskGAIA (val)
Level 1 Score56.6
97
Agentic ReasoningGAIA (val)
Average Score37.5
17
General Assistant TasksGAIA
Average Score46.06
16
SummarizationSummaryBench Dynasts--
9
Web ReasoningAssistantBench
Accuracy25.3
8
Hierarchical SummarizationSummaryBench Romeo and Juliet (R&J)
Fidelity Rate (FR)100
7
Hierarchical SummarizationSummaryBench The Dynasts
Fidelity Rate (FR)100
7
Structured Data ExtractionSchemaBench
Chemistry Str. Acc100
7
Structured data extraction and navigationSchemaBench
Chemistry FR80
7
Hierarchical SummarizationSummaryBench Decline and Fall of the Roman Empire
Fidelity Rate (FR)100
7
Showing 10 of 23 rows

Other info

Follow for update