GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation

TL;DR

GIVE tackles the limitation that LLM reasoning often relies solely on internal knowledge and can hallucinate when external data is sparse or noisy. It introduces a graph-inspired veracity extrapolation framework that builds entity groups from a knowledge graph, induces inner- and inter-group connections, and guides divergent reasoning via affirmative, counterfactual, and expert knowledge, with progressive answer generation. Across biomedical, commonsense, and open-domain tasks, GIVE improves accuracy across LLM sizes, enabling smaller models to surpass larger ones in scientific tasks and demonstrating training-free applicability with scalable knowledge graphs. The approach reduces hallucinations, provides interpretable reasoning, and shows strong potential for deployment in knowledge-intensive domains where comprehensive external databases are impractical.

Abstract

Existing approaches based on context prompting or reinforcement learning (RL) to improve the reasoning capacities of large language models (LLMs) depend on the LLMs' internal knowledge to produce reliable Chain-Of-Thought (CoT). However, no matter the size of LLMs, certain problems cannot be resolved in a single forward pass. Meanwhile, agent-based reasoning systems require access to a comprehensive nonparametric knowledge base, which is often costly or not feasible for use in scientific and niche domains. We present Graph Inspired Veracity Extrapolation (GIVE), a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input. GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak). Extensive experiments demonstrated the following benefits of our framework: (1) GIVE boosts the performance of LLMs across various sizes. (2) In some scenarios, GIVE allows smaller LLMs to surpass larger, more sophisticated ones in scientific tasks (GPT3.5T + GIVE > GPT4). (3) GIVE is effective on scientific and open-domain assessments. (4) GIVE is a training-free method that enables LLMs to tackle new problems that extend beyond their training data (up to 43.5% -> 88.2%} accuracy improvement). (5) GIVE allows LLM agents to reason using both restricted (very small) and noisy (very large) knowledge sources, accommodating knowledge graphs (KG) ranging from 135 to more than 840k nodes. (6) The reasoning process involved in GIVE is fully interpretable.

Figures (8)

• Figure 1: Without gold context, Chain-of-Thought (CoT) fails because LLM's internal knowledge fails to form a faithful logic chain; Retrieval-Augmented-Generation (RAG) retrieves semantically similar but irrelevant information, leading to hallucination; Think-on-Graph (ToG) focuses on using LLM to prune the direction of traverse, thus fails for lack of high-quality candidates.
• Figure 2: Reasoning process of GIVE. Solid lines indicate expert information, while dashed lines depict results from the "veracity extrapolation" procedure. GIVE first constructs an entity group for each queried concept, then induce inner-group connections using its internal knowledge. The expert's cross-group connections serve as evidence, guiding the LLM to extrapolate the veracity of potential relationships among similar concepts.
• Figure 3: GPT4o win rate on TruthfulQA truthfulqa in %.
• Figure 4: Different factors that may impact GIVE's performance: (upper) GIVE's performance VS no.additional entities per group. (lower) GIVE's performance VS expert knowledge ratio of the "veracity extrapolation" process
• Figure 5: Sensitivity analysis of GIVE on different number of seeded examples.