The Curious Case of Factual (Mis)Alignment between LLMs' Short- and Long-Form Answers

TL;DR

This paper introduces SLAQ, a benchmark that probes factual consistency in LLMs by comparing answers to the same facts when posed as short, isolated questions versus longer, integrated prompts. Across 16 models and 600 topics drawn from Wikipedia, the authors reveal systematic misalignment: short-form answers are typically more accurate, but long-form responses show position-dependent degradation and momentum effects that propagate errors. They provide mechanistic interpretability evidence, showing that aligned facts share more similar internal circuits, and demonstrate that six circuit-similarity metrics can predict alignment with ROC-AUC up to 0.85. The work challenges the assumption that good performance on simple factual questions guarantees reliability on complex knowledge tasks, and suggests benchmarking approaches and targeted interventions to improve factual consistency in open-domain QA.

Abstract

Large language models (LLMs) can correctly answer "When was Einstein born?" yet fail to provide the same date when writing about Einstein's life revealing a fundamental inconsistency in how models access factual knowledge across task complexities. While models display impressive accuracy on factual question-answering benchmarks, the reliability gap between simple and complex queries remains poorly understood, eroding their trustworthiness. In this work, we introduce Short-Long Form Alignment for Factual Question Answering (SLAQ), a controlled evaluation framework that compares LLMs' answers to the same factual questions asked (a) in isolation (short) vs. (b) integrated into complex queries (long). Looking at 16 LLMs across 600 queries, we find a systematic misalignment of answers to the corresponding short and long queries. We further uncover position-dependent accuracy loss and momentum effects where consecutive correct or incorrect answers create self-reinforcing patterns. Through mechanistic analysis, we find that aligned facts activate overlapping model internals, and that metrics based on mechanistic similarity can predict short-long answer alignment with up to 78% accuracy. Our work establishes factual consistency over query complexity as an important aspect of LLMs' trustworthiness and challenges current evaluation practices, which implicitly assume that good performance for simple factual queries implies reliability in more complex knowledge-seeking tasks too.

Figures (6)

• Figure 1: Illustration of our Short-Long Form Alignment for Factual Question Answering (SLAQ) framework. An instance in our SLAQ benchmark is a complex knowledge-seeking query, i.e., a long query, which consists of five simple factual sub-queries, i.e., short queries, each with an unambiguous correct answer. LLMs independently generate the answers to (1) the long query (i.e., all five short queries combined) and (2) each of the five short queries in isolation. We use a state-of-the-art commercial LLM to judge the correctness of the generated answers to both the long query and short queries against the set of reference answers; we use these judgments to compute models' short- and long-form accuracy ($F_{S}$, $F_{L}$) as well as the short-long alignment scores.
• Figure 2: Short–long factual alignment results across model families. (a) Factual Correctness: per-model short-form accuracy $F_S$ (green) and long-form accuracy $F_L$ (purple). (b) Alignment: $\mathrm{Align}$ = percentage of facts with the same correctness label in short vs. long responses. (c) Signed Alignment: average over topics; for a single topic, the score is the average of $\mathrm{Align}_{\pm}$ of its five facts. Models key: G = Gemma, L = Llama, Q = Qwen (e.g., Q3–8B-R = Qwen-3, 8B parameters, R - reasoning).
• Figure 3: Long-form QA dynamics by sub-fact position. (a) Slot accuracy: percent correct for each fact position (slots 1–5) in the LQ answer. (b) Trailing 1-streak: $P(\mathrm{correct})$ for the current slot (2–5), conditioned on the length of the immediately preceding run of correct slots. (c) Trailing 0-streak: $P(\mathrm{correct})$ for the current slot (2–5), conditioned on the length of the immediately preceding run of incorrect slots.
• Figure 4: Circuit similarity comparison between aligned and misaligned facts across six metrics. Aligned facts (green) show significantly (p < 0.001) higher mechanistic similarity than misaligned facts (red) for all measures.
• Figure 5: Predictive modeling performance using circuit similarity metrics. (a) Individual feature performance shows Spearman Attention as the strongest single predictor of factual alignment (ROC-AUC = 0.83). (b) Combined features achieve robust performance across all evaluation metrics (ROC-AUC = 0.81, Accuracy = 0.76). (c) Feature importance reveals Spearman Attention as the dominant predictor (coefficient = 1.36).