LogicScore: Fine-grained Logic Evaluation of Conciseness, Completeness, and Determinateness in Attributed Question Answering

TL;DR

This work introduces LogicScore, a Horn-rule–based framework that shifts AQA evaluation from local factual grounding to global reasoning coherence by measuring Completeness, Conciseness, and Determinateness. Through experiments on HotpotQA, MusiQue, and 2WikiMultiHopQA with over 20 LLMs, it reveals a pervasive attribution myopia where high factual grounding does not guarantee sound deductive reasoning, and uncovers a scaling paradox where larger models become more determinative yet less concise. The methodology combines Answer Generation with Chain-of-Thought, Logic Transformation into Horn clauses, and structured logic evaluation, yielding a robust standard that correlates well with human judgments. The findings advocate prioritizing logical coherence in LLM development to ensure reliable, verifiable, long-form reasoning beyond surface-level factual grounding.

Abstract

Current evaluation methods for Attributed Question Answering (AQA) suffer from \textit{attribution myopia}: they emphasize verification of isolated statements and their attributions but overlook the global logical integrity of long-form answers. Consequently, Large Language Models (LLMs) often produce factually grounded yet logically incoherent responses with elusive deductive gaps. To mitigate this limitation, we present \textsc{LogicScore}, a unified evaluation framework that shifts the paradigm from local assessment to global reasoning scrutiny. Grounded in Horn Rules, our approach integrates a backward verification mechanism to systematically evaluate three key reasoning dimensions: \textit{Completeness} (logically sound deduction), \textit{Conciseness} (non-redundancy), and \textit{Determinateness} (consistent answer entailment). Extensive experiments across three multi-hop QA datasets (HotpotQA, MusiQue, and 2WikiMultiHopQA) and over 20 LLMs (including GPT-5, Gemini-3-Pro, LLaMA3, and task-specific tuned models) reveal a critical capability gap: leading models often achieve high attribution scores (e.g., 92.85\% precision for Gemini-3 Pro) but struggle with global reasoning quality (e.g., 35.11\% Conciseness for Gemini-3 Pro). Our work establishes a robust standard for logical evaluation, highlighting the need to prioritize reasoning coherence alongside factual grounding in LLM development. Codes are available at: https://github.com/zhichaoyan11/LogicScore.

Figures (12)

• Figure 1: Motivation of LogicScore. Traditional methods (pink area) yield high Factual Quality by focusing on local evidence, overlooking reasoning flaws. Our framework (blue area) evaluates Logic Quality via Completeness, Conciseness, and Determinateness, exposing logical deficits in the long-form answer.
• Figure 2: Overview of the LogicScore evaluation framework, consisting of three phases: (1) Answer Generation, where the model produces a Long-form Answer ($\mathcal{LA}$) and a Short Answer ($\mathcal{SA}$) based on the question and top-$k$ documents; (2) Logic Transformation, which decomposes the $\mathcal{LA}$ into a set of atomic propositions ($\mathbb{P}$) structured as Horn clauses; and (3) Logic Evaluation, which assesses the reasoning quality across three dimensions: Completeness, Conciseness and Determinateness.
• Figure 3: Case study. We observe three logic error types when prompting LLMs to generate attributed long-form answers: Circular denotes self-referential reasoning; Deviated represents a fundamental divergence in the reasoning trajectory; Broken signifies a logical discontinuity in the deductive chain. In contrast, Connected marks a complete reasoning chain, while the icon symbolizes the ideal reasoning paradigm.
• Figure 4: Impact of reasoning depth on logic quality metrics. The plots illustrate the performance of Proprietary, Open-source, and SFT LLMs across varying hops (2, 3, and 4).
• Figure 5: scaling Influence in multi-hops.