A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering

TL;DR

This work introduces SARA, a dataset that frames statutory reasoning in US tax law by pairing simplified IRC rules with natural-language entailment and numerical questions. A Prolog-based symbolic solver demonstrates perfect accuracy on the task, highlighting the potential of explicit rule-based reasoning versus current ML approaches. Through a constructed legal text corpus, tax-specific vectors, and a legally tuned BERT, the paper interrogates whether contemporary NLP models can effectively leverage prescriptive legal rules, finding that standard baselines offer limited gains. The resource clarifies the gap between symbolic and statistical methods in legal reasoning, and it argues for data-efficient and domain-aware approaches to advance real-world statutory reasoning systems.

Abstract

Legislation can be viewed as a body of prescriptive rules expressed in natural language. The application of legislation to facts of a case we refer to as statutory reasoning, where those facts are also expressed in natural language. Computational statutory reasoning is distinct from most existing work in machine reading, in that much of the information needed for deciding a case is declared exactly once (a law), while the information needed in much of machine reading tends to be learned through distributional language statistics. To investigate the performance of natural language understanding approaches on statutory reasoning, we introduce a dataset, together with a legal-domain text corpus. Straightforward application of machine reading models exhibits low out-of-the-box performance on our questions, whether or not they have been fine-tuned to the legal domain. We contrast this with a hand-constructed Prolog-based system, designed to fully solve the task. These experiments support a discussion of the challenges facing statutory reasoning moving forward, which we argue is an interesting real-world task that can motivate the development of models able to utilize prescriptive rules specified in natural language.

Figures (3)

• Figure 1: Sample cases from our dataset. The questions can be answered by applying the rules contained in the statutes to the context.
• Figure 2: Resources. Corpora on the left hand side were used to build the datasets and models on the right hand side.
• Figure 3: Example predicates used.