SMADE-IE: Sparse Multi-Agent Framework with Evidence-Driven Debate for Zero-Shot Information Extraction
About
Zero-shot information extraction (IE) with large language models (LLMs) has attracted increasing attention due to its flexibility in adapting to new schemas and domains without task-specific training. Existing approaches mainly rely on monolithic prompting, each-type prompting, or multi-agent debate. However, monolithic prompting often suffers from boundary and type errors, while each-type prompting and multi-agent debate introduce cross-type conflicts, redundant agent interactions, and substantial token overhead. To address these challenges, we propose SMADE-IE, a sparse and evidence-driven multi-agent framework for zero-shot IE. SMADE-IE first employs an Adaptive Mode Selector to dynamically route inputs into either a lightweight Global Extraction Mode or a Type-Centric Extraction Mode, reducing unnecessary type selection and reasoning noise. For conflicting predictions, we further introduce an Evidence-Driven Debate mechanism that structures arguments into Toulmin-style components and performs confidence aggregation through external evidence scoring and Bayesian updates. Experimental results on 9 benchmark datasets across NER, RE, and JERE tasks show that SMADE-IE consistently outperforms existing zero-shot IE baselines while also improving token efficiency through sparse agent selection and early-stopping debate.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Named Entity Recognition | CoNLL 2003 (test) | -- | 556 | |
| Relation Extraction | SciERC | Relation Strict F111.2 | 99 | |
| Named Entity Recognition | OntoNotes 5.0 (test) | -- | 90 | |
| Joint Entity and Relation Extraction | NYT | -- | 38 | |
| Joint Entity and Relation Extraction | CONLL04 | Entity F162.09 | 35 | |
| Named Entity Recognition | CrossNER (test) | -- | 13 | |
| Relation Extraction | DocRED | F1 Score30.78 | 11 | |
| Relation Extraction | SemEval 2010 | F1 Precision30.6 | 10 | |
| Named Entity Recognition | REDFM (test) | F1p Score31.12 | 5 | |
| Relation Extraction | CrossRE | F1 Score (p)16.22 | 5 |