Challenges for Generative AI in Legal Reasoning
Eljas Linna, Tuula Linna
TL;DR
This paper investigates the reliability challenges of generative AI in legal reasoning, focusing on high-stakes judicial decision-making. It adopts the Issue-Rule-Application-Conclusion (IRAC) framework to organize challenges and maps AI mechanisms such as retrieval-augmented generation, neuro-symbolic AI, and multi-agent systems to address them. It proposes a domain-aware, four-category evaluation framework (normative, doctrinal, evidential, technical) with testable obligations and metrics, highlighting that current techniques solve only narrow tasks and that staged adoption is prudent. The work guides future research toward auditable decision chains, calibrated uncertainty handling, and hybrid architectures that respect legal hierarchies and temporality, aiming to enable broader, safer AI-assisted adjudication in the long term.
Abstract
Large Language Models (LLMs) are being integrated into professional domains, yet their limitations in such high-stakes fields as law remain poorly understood. In response, this paper introduces examples of critical challenges to the functioning of generative and other forms of artificial intelligence (AI) as reliable reasoning tools in judicial decision-making. The study deconstructs core requirements and challenges for AI, including the ability to select the correct legal framework across jurisdictions, generate sound arguments based on the doctrine of the sources of law, distinguish ratio decidendi and obiter dicta in case law, resolve ambiguity arising from general clauses like "reasonableness", manage conflicting legal provisions, and apply the burden of proof correctly. The paper maps various AI enhancement mechanisms, such as retrieval-augmented generation (RAG), multi-agent systems and neuro-symbolic AI, to these challenges, assessing their potential to bridge the gap between the probabilistic nature of LLMs and the rigorous, choice-driven demands of legal interpretation. Furthermore, the paper sketches a path towards an evaluation framework, proposing that legal requirements be organized into normative, doctrinal, evidential, and technical categories, and subsequently operationalized into domain-specific, testable design obligations. The findings indicate that these techniques can address specific narrow challenges, but they fail to solve the more significant ones, particularly in tasks requiring discretion and transparent, justifiable reasoning. Therefore, we advocate for a staged adoption, first capturing efficiency in simple cases with technology already available today and sustaining long-term investment in new methods that handle hierarchy, temporality, and other requirements of legally sound reasoning, thus enabling expansion to complex adjudication in the future.