Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ConSens: Assessing context grounding in open-book question answering

Ivan Vankov, Matyo Ivanov, Adriana Correia, Victor Botev

TL;DR

ConSens introduces a perplexity-based metric to quantify how strongly a provided context grounds an LLM's open-book QA output. By comparing token-level perplexities with and without context, and transforming the ratio into a [-1,1] score, ConSens efficiently signals context reliance without requiring expensive LLM-as-a-judge evaluations. Validation across WikiEval, biomedical full/partial contexts, and RAG settings shows strong discrimination between grounded and ungrounded answers, with robustness across evaluator models. The approach is lightweight, privacy-preserving, and adaptable to real-time evaluation, offering a practical tool for assessing grounding in knowledge-intensive tasks and beyond.

Abstract

Large Language Models (LLMs) have demonstrated considerable success in open-book question answering (QA), where the task requires generating answers grounded in a provided external context. A critical challenge in open-book QA is to ensure that model responses are based on the provided context rather than its parametric knowledge, which can be outdated, incomplete, or incorrect. Existing evaluation methods, primarily based on the LLM-as-a-judge approach, face significant limitations, including biases, scalability issues, and dependence on costly external systems. To address these challenges, we propose a novel metric that contrasts the perplexity of the model response under two conditions: when the context is provided and when it is not. The resulting score quantifies the extent to which the model's answer relies on the provided context. The validity of this metric is demonstrated through a series of experiments that show its effectiveness in identifying whether a given answer is grounded in the provided context. Unlike existing approaches, this metric is computationally efficient, interpretable, and adaptable to various use cases, offering a scalable and practical solution to assess context utilization in open-book QA systems.

ConSens: Assessing context grounding in open-book question answering

TL;DR

ConSens introduces a perplexity-based metric to quantify how strongly a provided context grounds an LLM's open-book QA output. By comparing token-level perplexities with and without context, and transforming the ratio into a [-1,1] score, ConSens efficiently signals context reliance without requiring expensive LLM-as-a-judge evaluations. Validation across WikiEval, biomedical full/partial contexts, and RAG settings shows strong discrimination between grounded and ungrounded answers, with robustness across evaluator models. The approach is lightweight, privacy-preserving, and adaptable to real-time evaluation, offering a practical tool for assessing grounding in knowledge-intensive tasks and beyond.

Abstract

Large Language Models (LLMs) have demonstrated considerable success in open-book question answering (QA), where the task requires generating answers grounded in a provided external context. A critical challenge in open-book QA is to ensure that model responses are based on the provided context rather than its parametric knowledge, which can be outdated, incomplete, or incorrect. Existing evaluation methods, primarily based on the LLM-as-a-judge approach, face significant limitations, including biases, scalability issues, and dependence on costly external systems. To address these challenges, we propose a novel metric that contrasts the perplexity of the model response under two conditions: when the context is provided and when it is not. The resulting score quantifies the extent to which the model's answer relies on the provided context. The validity of this metric is demonstrated through a series of experiments that show its effectiveness in identifying whether a given answer is grounded in the provided context. Unlike existing approaches, this metric is computationally efficient, interpretable, and adaptable to various use cases, offering a scalable and practical solution to assess context utilization in open-book QA systems.
Paper Structure (12 sections, 3 equations, 3 figures, 3 tables)

This paper contains 12 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Contribution
  3. Related work
  4. ConSens
  5. Validation
  6. Experiment 1: Evaluation of grounded versus ungrounded answers.
  7. Experiment 2: Evaluation of full versus partial context.
  8. Experiment 3: Context grounding in RAG setting.
  9. Comparison with other metrics
  10. Evaluator model
  11. Discussion
  12. Conclusions

Figures (3)

  • Figure 1: Experiment 1 results. The x-axis represents the ConSens value for the grounded answer, while the y-axis corresponds to the ConSens value for the ungrounded answer. The percentage values indicate the proportion of scores that fall within the respective ranges of the two ConSens scores.
  • Figure 2: Experiment 2 results. Values on the x axis represent the ConSens value when the full context was provided and the y axis represents the ConSens scores in the partial context condition.
  • Figure 3: Experiment 3 results. The x axis represents the minimum value of ConSens when the correct document (i.e. the abstract which was used to generate the question) was included in the context. Accordingly, the y axis stands for the value of ConSens when the correct document was not part of the context.