Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Chatlaw: A Multi-Agent Legal Assistant based on a Role-Aligned Mixture-of-Experts Architecture

About

Artificial Intelligence (AI) holds great potential in legal services, yet Large Language Models (LLMs) face two major challenges: limited knowledge of the Chinese legal system and vulnerability to hallucinations. To address these issues, we present Chatlaw, a multi-agent legal assistant. Chatlaw's framework is designed to emulate the Standard Operating Procedures (SOP) of real law firms, where different roles (e.g., assistant, researcher, senior lawyer) collaborate on a case. To computationally mirror this collaborative structure, we developed a novel Role-Aligned Mixture-of-Experts (RA-MoE) architecture. In this system, the internal "experts" are specifically trained to align with the distinct tasks of each agent role (e.g., inquiry, analysis, drafting). These specialized agents (Legal Assistant, Researcher, etc.) then form the collaborative framework. When they interact with users, retrieve legal knowledge, analyze case details, or generate reliable consultations, the RA-MoE architecture intelligently routes their computations to the corresponding dedicated expert, ensuring each step is handled by the most qualified parameters. In evaluations, Chatlaw surpasses general-purpose AI models, including GPT-4, achieving a 7.73% improvement in accuracy on the LawBench benchmark and an 11-point higher score on the Unified Qualification Exam for Legal Professionals. Real-case studies and expert assessments further confirm its robustness. Chatlaw enhances the accessibility and reliability of legal services, advancing the provision of legal support to the public.

Jiaxi Cui, Munan Ning, Zongjian Li, Bohua Chen, Yang Yan, Hao Li, Bin Ling, Yonghong Tian, Li Yuan• 2023

Related benchmarks

TaskDatasetResultRank
Legal ReasoningLexEval
Memoization16.31
35
Legal Knowledge and Reasoning BenchmarkLawBench
Memorization Score43.86
30
Legal Question AnsweringLawBench revised (test)
ROUGE-127.78
17
Knowledge QuestioningJ1 EVAL
Average Score52.9
14
Legal ConsultationJ1 EVAL
Average Score36.5
14
Defence DraftingJ1 EVAL
FOR15.8
14
Civil CourtJ1 EVAL
PFS Score3.7
14
Complaint DraftingJ1 EVAL
FOR Score30.3
14
Criminal CourtJ1 EVAL
PFS (Procedural Fairness Score)3.7
14
Political and Legal Affairs AssessmentPoliLegal
Average Score51.25
14
Showing 10 of 14 rows

Other info

Follow for update