Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models

TL;DR

The paper frames factual queries for LLMs as constraint satisfaction problems and introduces SAT Probe, an attention-based predictor that estimates constraint satisfaction from the model's internal attention to constraint tokens. Through a 10-dataset benchmark spanning 40k prompts and Llama-2 scales, it demonstrates a robust link between constraint-focused attention and factual accuracy, enabling per-constraint diagnostics and potential early stopping to save computation. The approach yields AUROC comparable to model confidence and provides fine-grained feedback that can guide reliability improvements. This mechanistic perspective advances understanding of when and why LLMs hallucinate and offers practical tools for monitoring and mitigating factual errors in real time.

Abstract

We investigate the internal behavior of Transformer-based Large Language Models (LLMs) when they generate factually incorrect text. We propose modeling factual queries as constraint satisfaction problems and use this framework to investigate how the LLM interacts internally with factual constraints. We find a strong positive relationship between the LLM's attention to constraint tokens and the factual accuracy of generations. We curate a suite of 10 datasets containing over 40,000 prompts to study the task of predicting factual errors with the Llama-2 family across all scales (7B, 13B, 70B). We propose SAT Probe, a method probing attention patterns, that can predict factual errors and fine-grained constraint satisfaction, and allow early error identification. The approach and findings take another step towards using the mechanistic understanding of LLMs to enhance their reliability.

Figures (25)

• Figure 1: Tracking attention to predict constraint satisfaction and factual errors. We consider factual queries to LLMs as constraint satisfaction problems i.e., factual queries impose a set of constraints that the LLM’s responses must satisfy. To predict constraint satisfaction (i.e., factual correctness), we track the attention to the constraint tokens in an LLM (here, Llama-2 13B). We find that attention to the constraint tokens highly correlates with factual correctness. The red (resp. blue) text indicates factually incorrect (resp. correct) completions.
• Figure 2: Difficulty of the factual query vs LLM performance. Left: Popularity vs Correctness We observe that the more popular the entity in the factual query is, the more correct the LLMs are. Right: Constrainedness vs Correctness We observe that the more constrained the problem is (i.e., has a smaller set of potential solutions), the less correct the LLMs are.
• Figure 3: Tracking attention to predict factual errors in single-constraint settings. We track the attention contribution from the constraint tokens during generation. We observe a small-norm contribution ($||{\mathbf{a}}_{i, T}^{\ell}||$) when the LLM makes a factual error and a larger-norm attention contribution when the LLM is factually correct. The red text indicates factually incorrect completions, whereas the blue text indicates factually correct completions.
• Figure 4: Attention correlates with correctness. The first two columns of panels give the $25$ samples for which the LLM makes the most and the least confident predictions. The color indicates the norm of the attention contribution from the constraint, where each column in the panel captures a layer in the LLM and each row is a specific sample. The last column of panels relates the total attention to constraints and accuracy, where the x-axis is the attention contribution percentile in the dataset and the y-axis is the accuracy in the bin. The results are for the year of birth queries (see \ref{['appendix:basketball_players-prompt']}).
• Figure 5: Attention contribution and model scaling. Here, the x-axis and y-axis show the attention to the constraints for the smaller LLM and the larger LLM, respectively, and are normalized by dividing by the maximum value of total attention in the dataset. Coloring is determined by which of the two LLMs succeeds in factual queries. We group the factual queries by their x-axis value and y-axis values and color the cell with the most frequent category in the cell. Appendix Figure \ref{['fig:appendix-scaling']} presents the complete scatter plot.

Theorems & Definitions (1)

• Definition 3.1: Factual Query as a CSP