To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts

TL;DR

This work defines situated faithfulness for LLMs when using external context and proposes two complementary strategies: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). It introduces a new RedditQA dataset to test robustness against natural incorrect contexts and demonstrates that SCR generally outperforms RCR on strong reasoning models, while RCR benefits smaller models. A novel training method, Confidence Reasoning Direct Preference Optimization (CR-DPO), further enhances SCR and improves generalization, particularly for Llama-3-8B. The results offer practical guidelines for deploying context-aware QA systems and point to promising directions for future research in long-form QA and multi-context settings.

Abstract

Large Language Models (LLMs) are often augmented with external contexts, such as those used in retrieval-augmented generation (RAG). However, these contexts can be inaccurate or intentionally misleading, leading to conflicts with the model's internal knowledge. We argue that robust LLMs should demonstrate situated faithfulness, dynamically calibrating their trust in external information based on their confidence in the internal knowledge and the external context to resolve knowledge conflicts. To benchmark this capability, we evaluate LLMs across several QA datasets, including a newly created dataset featuring in-the-wild incorrect contexts sourced from Reddit posts. We show that when provided with both correct and incorrect contexts, both open-source and proprietary models tend to overly rely on external information, regardless of its factual accuracy. To enhance situated faithfulness, we propose two approaches: Self-Guided Confidence Reasoning (SCR) and Rule-Based Confidence Reasoning (RCR). SCR enables models to self-assess the confidence of external information relative to their own internal knowledge to produce the most accurate answer. RCR, in contrast, extracts explicit confidence signals from the LLM and determines the final answer using predefined rules. Our results show that for LLMs with strong reasoning capabilities, such as GPT-4o and GPT-4o mini, SCR outperforms RCR, achieving improvements of up to 24.2% over a direct input augmentation baseline. Conversely, for a smaller model like Llama-3-8B, RCR outperforms SCR. Fine-tuning SCR with our proposed Confidence Reasoning Direct Preference Optimization (CR-DPO) method improves performance on both seen and unseen datasets, yielding an average improvement of 8.9% on Llama-3-8B. In addition to quantitative results, we offer insights into the relative strengths of SCR and RCR.

Figures (3)

• Figure 1: Concept illustration of Self-guided Confidence Reasoning and Rule-based Confidence Reasoning. The goal is to enable the model to generate correct answers, regardless of input context accuracy, in contrast to direct input augmentation, which often blindly follows the context.
• Figure 2: An example where SCR fails to maintain situated faithfulness, but CR-DPO succeeds. In this case, the model’s internal answer is correct, but the context is misleading. Before training with CR-DPO, the model is misled despite correctly listing factual details. After training, it successfully arrives at the correct answer.
• Figure 3: Impact of Thresholds on Situated Faithfulness for InternalConf Using GPT-4o-mini