PANORAMA: A Dataset and Benchmarks Capturing Decision Trails and Rationales in Patent Examination

TL;DR

PANORAMA introduces a large-scale, claim-level dataset of USPTO patent examination records that preserves examiner decision trails and rationales. It decomposes the examination process into three sequential benchmarks—prior-art retrieval (PAR4PC), paragraph identification (PI4PC), and novelty/non-obviousness classification (NOC4PC)—to evaluate LLMs at each step of patent review. Baseline results show that current models retrieve relevant art but struggle with robust novelty and non-obviousness judgments, while supervised fine-tuning on PANORAMA yields consistent improvements. The dataset enables more realistic modeling of patent examination and motivates future work to better capture expert reasoning and cross-jurisdictional variations, with data publicly available for reproducibility.

Abstract

Patent examination remains an ongoing challenge in the NLP literature even after the advent of large language models (LLMs), as it requires an extensive yet nuanced human judgment on whether a submitted claim meets the statutory standards of novelty and non-obviousness against previously granted claims -- prior art -- in expert domains. Previous NLP studies have approached this challenge as a prediction task (e.g., forecasting grant outcomes) with high-level proxies such as similarity metrics or classifiers trained on historical labels. However, this approach often overlooks the step-by-step evaluations that examiners must make with profound information, including rationales for the decisions provided in office actions documents, which also makes it harder to measure the current state of techniques in patent review processes. To fill this gap, we construct PANORAMA, a dataset of 8,143 U.S. patent examination records that preserves the full decision trails, including original applications, all cited references, Non-Final Rejections, and Notices of Allowance. Also, PANORAMA decomposes the trails into sequential benchmarks that emulate patent professionals' patent review processes and allow researchers to examine large language models' capabilities at each step of them. Our findings indicate that, although LLMs are relatively effective at retrieving relevant prior art and pinpointing the pertinent paragraphs, they struggle to assess the novelty and non-obviousness of patent claims. We discuss these results and argue that advancing NLP, including LLMs, in the patent domain requires a deeper understanding of real-world patent examination. Our dataset is openly available at https://huggingface.co/datasets/LG-AI-Research/PANORAMA.

Figures (6)

• Figure 1: The overall framework of the PANORAMA dataset and benchmark construction. The PANORAMA dataset is constructed from documents appearing in patent examination. It comprises patent documents and office actions, such as non-final rejections. The benchmark tasks are designed to emulate the sequential nature of the patent examination process.
• Figure 2: An overview of the PANORAMA dataset curation pipeline.
• Figure 3: Distribution of Claims Count in USPTO Patent Applications. The three panels show the frequency distribution of: (Left) initial claims count in original patent applications, (Center) final claims count including canceled claims during prosecution, and (Right) final claims count excluding canceled claims.
• Figure 4: Distribution of rejections by section code (left) and IPC categories (right) in the PANORAMA dataset.
• Figure 5: Distribution of cited patents count (left) and rejection rates (right) in the PANORAMA dataset.