PILOT-Bench: A Benchmark for Legal Reasoning in the Patent Domain with IRAC-Aligned Classification Tasks

TL;DR

PILOT-Bench introduces a PTAB-centric benchmark that couples PTAB ex parte decisions with USPTO patent data to evaluate large language models' capacity for patent-domain legal reasoning. It operationalizes the IRAC framework into three classification tasks—Issue Type, Board Authorities, and Subdecision—calibrated to real PTAB practice and evaluated across multiple model families under zero-shot settings. Empirical results show closed-source models consistently outperform open-source ones, while long-tail label distributions and input design significantly affect performance, highlighting gaps in current reasoning capabilities. The benchmark provides a durable dataset and protocol to quantify and drive improvements in patent-domain legal reasoning, with future work targeting generation-based applications and broader PTAB contexts to enable more robust, responsible deployment.

Abstract

The Patent Trial and Appeal Board (PTAB) of the USPTO adjudicates thousands of ex parte appeals each year, requiring the integration of technical understanding and legal reasoning. While large language models (LLMs) are increasingly applied in patent and legal practice, their use has remained limited to lightweight tasks, with no established means of systematically evaluating their capacity for structured legal reasoning in the patent domain. In this work, we introduce PILOT-Bench, the first PTAB-centric benchmark that aligns PTAB decisions with USPTO patent data at the case-level and formalizes three IRAC-aligned classification tasks: Issue Type, Board Authorities, and Subdecision. We evaluate a diverse set of closed-source (commercial) and open-source LLMs and conduct analyses across multiple perspectives, including input-variation settings, model families, and error tendencies. Notably, on the Issue Type task, closed-source models consistently exceed 0.75 in Micro-F1 score, whereas the strongest open-source model (Qwen-8B) achieves performance around 0.56, highlighting a substantial gap in reasoning capabilities. PILOT-Bench establishes a foundation for the systematic evaluation of patent-domain legal reasoning and points toward future directions for improving LLMs through dataset design and model alignment. All data, code, and benchmark resources are available at https://github.com/TeamLab/pilot-bench.

Figures (27)

• Figure 1: PILOT-Bench: Data sources, processing pipeline, and tasks. PTAB metadata JSONs and decision JSONs are aligned with USPTO patent JSONs to form PILOT-Bench (18K). From this base, we map each case to the appellant’s patent and apply an LLM opinion split, yielding the 15K Opinion Split Data used for IRAC-aligned classification tasks.
• Figure 2: Opinion Split of PTAB Decisions. Given a PTAB decision, an LLM segments the text at the sentence-level and, using context, classifies each sentence into four roles; appellant_arguments, examiner_findings, ptab_opinion, and facts. The resulting Opinion Split Data serves as the base input for our IRAC-aligned classification tasks.
• Figure 3: Label distributions across tasks are imbalanced; for Subdecision (fine), only the top 10 labels are shown. Bold values under the labels are the proportion each label occupies in the dataset.
• Figure 4: Task-specific prompting. A standardized prompt combines a task-specific instruction with the appellant_arguments and examiner_findings segments; the LLM then executes the chosen task--Issue, Board Authorities, or Subdecision--and outputs from the predefined label set.
• Figure 5: PTAB decisions by year and subproceeding type (2007–2024).