From LSAT: The Progress and Challenges of Complex Reasoning

TL;DR

The paper investigates three LSAT-based complex-reasoning tasks—analytical reasoning, logical reasoning, and reading comprehension—via a hybrid architecture that combines symbolic, neural, and neural-symbolic components. It introduces AR-specific approaches (ARM, CGAR, NSAR), a logic-driven LR system (LReasoner) with context extension and data augmentation, and a position-aware RC model (P-DUMA) with transfer learning from RACE, reporting competitive results and insights into their respective strengths and failure modes. The study provides a detailed error analysis, demonstrates that symbolic and structured reasoning can outperform purely neural baselines under data-scarce conditions, and outlines key future directions, including unsupervised symbolic extraction, interpretability, few-shot learning, and all-encompassing benchmarks. Together, these components advance understanding of domain-general complex reasoning and lay groundwork for more robust, explainable AI systems capable of passing high-stakes standardized assessments.

Abstract

Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretation of logical knowledge and complex rule application. In this paper, we take a step forward in complex reasoning by systematically studying the three challenging and domain-general tasks of the Law School Admission Test (LSAT), including analytical reasoning, logical reasoning and reading comprehension. We propose a hybrid reasoning system to integrate these three tasks and achieve impressive overall performance on the LSAT tests. The experimental results demonstrate that our system endows itself a certain complex reasoning ability, especially the fundamental reading comprehension and challenging logical reasoning capacities. Further analysis also shows the effectiveness of combining the pre-trained models with the task-specific reasoning module, and integrating symbolic knowledge into discrete interpretable reasoning steps in complex reasoning. We further shed a light on the potential future directions, like unsupervised symbolic knowledge extraction, model interpretability, few-shot learning and comprehensive benchmark for complex reasoning.

Figures (5)

• Figure 1: Three examples of LSAT tasks with the required reasoning processes. For AR, it needs to understand the knowledge of participants, positions, and constraints, and deduce the legitimate option. For LR, the elementary logical symbols and expressions need to be identified to logically infer the implicit expression. For RC, it requires locating the relevant pieces by the positional indicators and abstract the answer. The options in green mean the correct answers of three examples.
• Figure 2: The overall illustration of our investigation from LSAT towards complex reasoning.
• Figure 3: The tree-based reasoning process. Here "T/F/-" means "True/False/Unknown".
• Figure 4: The overall architecture of the logic-driven context extension framework for LR. $c$, $q$, $o_i$ and $e_i$ are the context, question, $i$-th option and the extended context for $i$-th option, respectively. The texts in green mean that the option $B$ is matched against its extended context which has the highest score.
• Figure 5: An example of our annotation for an AR problem. C-$i$ means the $i$-th constraint while Q-$j$ denotes the pair of the question and option $j$.