Parameterized Argumentation-based Reasoning Tasks for Benchmarking Generative Language Models

TL;DR

This work addresses the challenge of evaluating the reasoning capabilities of generative language models in legal contexts by proposing parameterized, dynamically generated benchmarks based on abstract argumentation frameworks $(\mathcal{A}, \mathcal{E})$. It constructs linear and non-linear argument attack graphs, translates them into natural-language prompts via an ontology, and tests seven contemporary models, revealing brittleness and inconsistency in reasoning that intensify with task complexity. The key contributions are the formalized benchmark design, the ontology-driven prompt pipeline, and a comprehensive evaluation showing that even reasoning-specialized models struggle on higher-complexity non-linear prompts. The approach offers a path toward more robust, responsible AI systems in the legal domain by enabling scalable, contamination-resistant benchmarking and guiding future improvements in reasoning capabilities.

Abstract

Generative large language models as tools in the legal domain have the potential to improve the justice system. However, the reasoning behavior of current generative models is brittle and poorly understood, hence cannot be responsibly applied in the domains of law and evidence. In this paper, we introduce an approach for creating benchmarks that can be used to evaluate the reasoning capabilities of generative language models. These benchmarks are dynamically varied, scalable in their complexity, and have formally unambiguous interpretations. In this study, we illustrate the approach on the basis of witness testimony, focusing on the underlying argument attack structure. We dynamically generate both linear and non-linear argument attack graphs of varying complexity and translate these into reasoning puzzles about witness testimony expressed in natural language. We show that state-of-the-art large language models often fail in these reasoning puzzles, already at low complexity. Obvious mistakes are made by the models, and their inconsistent performance indicates that their reasoning capabilities are brittle. Furthermore, at higher complexity, even state-of-the-art models specifically presented for reasoning capabilities make mistakes. We show the viability of using a parametrized benchmark with varying complexity to evaluate the reasoning capabilities of generative language models. As such, the findings contribute to a better understanding of the limitations of the reasoning capabilities of generative models, which is essential when designing responsible AI systems in the legal domain.

Figures (11)

• Figure 1: Example of a linear argument attack graph and generated prompt
• Figure 2: Example of a non-linear argument attack graph and an generated prompt
• Figure 3: The pipeline of our approach for generating dynamic benchmarks of scaling complexity. In this paper, the graph parameters are the number of arguments in the graph, and the ontology is a list of names and statements
• Figure 4: The mean MCC of models on prompts that are shuffled and not shuffled based on non-linear attack graphs
• Figure 5: Linear attack graphs: percentage of correct answers versus the number of arguments in the prompt