DeonticBench: A Benchmark for Reasoning over Rules

Abstract

Reasoning with complex, context-specific rules remains challenging for large language models (LLMs). In legal and policy settings, this manifests as deontic reasoning: reasoning about obligations, permissions, and prohibitions under explicit rules. While many recent benchmarks emphasize short-context mathematical reasoning, fewer focus on long-context, high-stakes deontic reasoning. To address this gap, we introduce DEONTICBENCH, a benchmark of 6,232 tasks across U.S. federal taxes, airline baggage policies, U.S. immigration administration, and U.S. state housing law. These tasks can be approached in multiple ways, including direct reasoning in language or with the aid of symbolic computation. Besides free-form chain-of-thought reasoning, DEONTICBENCH enables an optional solver-based workflow in which models translate statutes and case facts into executable Prolog, leading to formal problem interpretations and an explicit program trace. We release reference Prolog programs for all instances. Across frontier LLMs and coding models, best hard-subset performance reaches only 44.4% on SARA Numeric and 46.6 macro-F1 on Housing. We further study training with supervised fine-tuning and reinforcement learning for symbolic program generation. Although training improves Prolog generation quality, current RL methods still fail to solve these tasks reliably. Overall, DEONTICBENCH provides a benchmark for studying context-grounded rule reasoning in real-world domains under both symbolic and non-symbolic settings.

Figures (4)

• Figure 1: Walkthrough of a DeonticBench instance in the symbolic setting. (1) Given the full problem context, the model performs deontic reasoning to identify and apply the relevant rules. (2) The LLM translates the problem into Prolog code. (3) The generated Prolog is executed by SWI-Prolog solver. The illustrated example is a 2017 tax-liability case.
• Figure 2: Performance decomposition analysis for SARA Numeric and Airline. Each model shows three bars (left to right: Direct, Zero-Shot, Few-Shot). Plots for other domains are in Figure \ref{['fig:error-analysis-appendix']} (Appendix \ref{['appendix:additional-error-bar']}).
• Figure 3: Reasoning effort ablation on SARA Numeric hard cases. Each panel corresponds to a different prompting strategy (Direct, Zero-Shot, and Few-Shot). Bars indicate mean accuracy, with error bars showing 95% bootstrap confidence intervals.
• Figure 4: Performance decomposition analysis for SARA Binary, Housing, and USCIS. Each model shows three bars (left to right: Direct, Zero-Shot, Few-Shot). The colored portion represents correct predictions; the solid pink portion represents wrong answers; the hatched pink portion represents abstentions (Prolog timeout or crash).