Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Missing vs. Unused Knowledge Hypothesis for Language Model Bottlenecks in Patent Understanding

Siyang Wu, Honglin Bao, Nadav Kunievsky, James A. Evans

TL;DR

The paper tackles the gap between knowledge storage and practical deployment in large language systems by focusing on patent understanding that requires fine-grained differentiation. It introduces a differentiation-based task and a diagnostic framework to separate missing from unused lay-in knowledge, backed by a large-scale USPTO patent dataset. Through Retrieval-Augmented Generation and self-questioning prompts, it shows that deployment limitations—not gaps in knowledge—predominate, with smaller models providing cues and larger models delivering deeper reasoning. The findings advocate for dynamic knowledge activation and scale-complementary strategies to advance domain-specific reasoning in LLMs, offering a shift from static recall to real-time knowledge deployment in complex, technical texts.

Abstract

While large language models (LLMs) excel at factual recall, the real challenge lies in knowledge application. A gap persists between their ability to answer complex questions and their effectiveness in performing tasks that require that knowledge. We investigate this gap using a patent classification problem that requires deep conceptual understanding to distinguish semantically similar but objectively different patents written in dense, strategic technical language. We find that LLMs often struggle with this distinction. To diagnose the source of these failures, we introduce a framework that decomposes model errors into two categories: missing knowledge and unused knowledge. Our method prompts models to generate clarifying questions and compares three settings -- raw performance, self-answered questions that activate internal knowledge, and externally provided answers that supply missing knowledge (if any). We show that most errors stem from failures to deploy existing knowledge rather than from true knowledge gaps. We also examine how models differ in constructing task-specific question-answer databases. Smaller models tend to generate simpler questions that they, and other models, can retrieve and use effectively, whereas larger models produce more complex questions that are less effective, suggesting complementary strengths across model scales. Together, our findings highlight that shifting evaluation from static fact recall to dynamic knowledge application offers a more informative view of model capabilities.

Missing vs. Unused Knowledge Hypothesis for Language Model Bottlenecks in Patent Understanding

TL;DR

The paper tackles the gap between knowledge storage and practical deployment in large language systems by focusing on patent understanding that requires fine-grained differentiation. It introduces a differentiation-based task and a diagnostic framework to separate missing from unused lay-in knowledge, backed by a large-scale USPTO patent dataset. Through Retrieval-Augmented Generation and self-questioning prompts, it shows that deployment limitations—not gaps in knowledge—predominate, with smaller models providing cues and larger models delivering deeper reasoning. The findings advocate for dynamic knowledge activation and scale-complementary strategies to advance domain-specific reasoning in LLMs, offering a shift from static recall to real-time knowledge deployment in complex, technical texts.

Abstract

While large language models (LLMs) excel at factual recall, the real challenge lies in knowledge application. A gap persists between their ability to answer complex questions and their effectiveness in performing tasks that require that knowledge. We investigate this gap using a patent classification problem that requires deep conceptual understanding to distinguish semantically similar but objectively different patents written in dense, strategic technical language. We find that LLMs often struggle with this distinction. To diagnose the source of these failures, we introduce a framework that decomposes model errors into two categories: missing knowledge and unused knowledge. Our method prompts models to generate clarifying questions and compares three settings -- raw performance, self-answered questions that activate internal knowledge, and externally provided answers that supply missing knowledge (if any). We show that most errors stem from failures to deploy existing knowledge rather than from true knowledge gaps. We also examine how models differ in constructing task-specific question-answer databases. Smaller models tend to generate simpler questions that they, and other models, can retrieve and use effectively, whereas larger models produce more complex questions that are less effective, suggesting complementary strengths across model scales. Together, our findings highlight that shifting evaluation from static fact recall to dynamic knowledge application offers a more informative view of model capabilities.
Paper Structure (18 sections, 11 figures)

This paper contains 18 sections, 11 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Hard-to-Distinguish Patents
  4. Evaluating LLMs’ Understanding of the Technology Domain
  5. Experimental setups
  6. Results
  7. Missing vs. Unused Knowledge
  8. It's Not Only What to Ask, but Who Asks It
  9. Final Remarks
  10. Understanding Through Differentiation
  11. Related Literature
  12. Judging Close Patents VS. Rewritten Patents
  13. The Patent Dataset
  14. Prompts
  15. Results for Majority Voting Schema
  16. ...and 3 more sections

Figures (11)

  • Figure 1: Panel (a): Model perplexity of LLaMA-3.1-8B-Instruct on paper texts and patents descriptions. The texts of patents are much more confusing to LLMs compared to scientific papers. Panel (b): Histogram of the distribution of each patent's maximum cosine similarity to another (distinct) patent.
  • Figure 2: Blue bars always represent the answer "yes", orange bars always represent "no". Top row: Ask LLMs "Do you think they are different patents?", given that they are actually different. Blue: "yes" (they are different). Orange: "no" (they are the same). Bottom row: "Do you think they are the same patent?", given that they are actually different. Blue: "yes" (they are the same). Orange: "no" (they are different). Llama models are instruction-tuning versions.
  • Figure 3: Panel a: The contribution to understanding decreases in the following order: scientific QA, self-QA, and question-only. $p$-values: $p_{q_1}$ to $p_{q_5}$ = 0.000, $p_{q_6}$ = 0.013 (self vs. scientific answers). Panel b: no significant difference could be found between scientific information and self-produced answers.
  • Figure 4: Question models: llama 1b, 8b, 70b, 405b. Answer models: 3b and 8b. Panels a and b: Self-generated questions are compatible with the model’s own answering capacity. Questions generated by smaller models are compatible with larger answer models but not the converse. "Compatible" means a lower misjudgment rate (judging as "same patents"). Panels c to h: mechanisms. Panels c and d: higher perplexity when answering small models' questions. Panels e and f: lower similarity between [small models' questions + answer models' answers] and [the patent texts], implying searching for more new information when answering small models' questions. Panels g and h: smaller models' questions are more open-ended (the SD of embedding distances of each answer from the mean of all answers).
  • Figure 5: Few-shot CoT reasoning can distinguish the rephrased patent texts (rewritten by the GPT-o3 model), as shown in the right panel (lower misjudge rate), but cannot distinguish their counterpart (another unique patent, as shown in the left panel). Y-axis always represents the response of "yes", i.e., the error rate (Left: yes when asking "are they the same" for close-but-different patent pairs. Right: yes when asking "are they different" for rephrased patents.)
  • ...and 6 more figures