Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Aristotle: Mastering Logical Reasoning with A Logic-Complete Decompose-Search-Resolve Framework

About

In the context of large language models (LLMs), current advanced reasoning methods have made impressive strides in various reasoning tasks. However, when it comes to logical reasoning tasks, major challenges remain in both efficacy and efficiency. This is rooted in the fact that these systems fail to fully leverage the inherent structure of logical tasks throughout the reasoning processes such as decomposition, search, and resolution. To address this, we propose a logic-complete reasoning framework, Aristotle, with three key components: Logical Decomposer, Logical Search Router, and Logical Resolver. In our framework, symbolic expressions and logical rules are comprehensively integrated into the entire reasoning process, significantly alleviating the bottlenecks of logical reasoning, i.e., reducing sub-task complexity, minimizing search errors, and resolving logical contradictions. The experimental results on several datasets demonstrate that Aristotle consistently outperforms state-of-the-art reasoning frameworks in both accuracy and efficiency, particularly excelling in complex logical reasoning scenarios. We will open-source all our code at https://llm-symbol.github.io/Aristotle/.

Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, William Yang Wang, Preslav Nakov, Mong-Li Lee, Wynne Hsu• 2024

Related benchmarks

TaskDatasetResultRank
Logical reasoningRepublicQA
Accuracy82.5
16
Reasoning accuracyPRONTOQA 5-hop
Accuracy94.8
14
Logical reasoningProofWriter depth-5 OWA setting
Accuracy (ProofWriter d5 OWA)63.23
8
Logical reasoningProverQA hard split
Accuracy0.562
8
Logical reasoningFOLIO full expert-curated
Accuracy68.68
8
Showing 5 of 5 rows

Other info

Follow for update