From Query to Counsel: Structured Reasoning with a Multi-Agent Framework and Dataset for Legal Consultation
About
Legal consultation question answering (Legal CQA) presents unique challenges compared to traditional legal QA tasks, including the scarcity of high-quality training data, complex task composition, and strong contextual dependencies. To address these, we construct JurisCQAD, a large-scale dataset of over 43,000 real-world Chinese legal queries annotated with expert-validated positive and negative responses, and design a structured task decomposition that converts each query into a legal element graph integrating entities, events, intents, and legal issues. We further propose JurisMA, a modular multi-agent framework supporting dynamic routing, statutory grounding, and stylistic optimization. Combined with the element graph, the framework enables strong context-aware reasoning, effectively capturing dependencies across legal facts, norms, and procedural logic. Trained on JurisCQAD and evaluated on a refined LawBench, our system significantly outperforms both general-purpose and legal-domain LLMs across multiple lexical and semantic metrics, demonstrating the benefits of interpretable decomposition and modular collaboration in Legal CQA.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Legal Question Answering | LawBench revised (test) | ROUGE-144.68 | 17 | |
| Cross-jurisdiction Legal Reasoning | KoBLEX | Token F1 Score34.88 | 11 | |
| Reading Comprehension | LawBench 2-5 | RC-F146.62 | 8 | |
| Cross-lingual Question Answering | LEGALBENCH RuleQA English (test) | ROUGE-120.25 | 3 | |
| Legal Consultation | LawBench Revised | Legal Soundness4.14 | 3 |