Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models

Delip Rao, Feijiang Han, Chris Callison-Burch

Abstract

We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0\% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.

ThinknCheck: Grounded Claim Verification with Compact, Reasoning-Driven, and Interpretable Models

Abstract

We present ThinknCheck, a 1B-parameter verifier for grounded claim verification that first produces a short, structured rationale and then a binary verdict. We construct LLMAggreFact-Think, a 24.1k reasoning-augmented training set derived from LLMAggreFact, and fine-tune a 4-bit Gemma3 model to follow this format. On LLMAggreFact, ThinknCheck attains 78.1 balanced accuracy (BAcc), surpassing MiniCheck-7B (77.4) with 7x fewer parameters; removing the reasoning step reduces BAcc to 57.5. On SciFact, ThinknCheck reaches 64.7 BAcc, a +14.7 absolute gain over MiniCheck-7B. By contrast, zero-shot chain-of-thought on the base Gemma3-1B harms accuracy relative to direct answers, and preference optimization with a simple format+accuracy reward underperforms supervised reasoning. To probe the latter, we introduce GSMClaims and a domain-specialized variant, ThinknCheck-Science, which improves across benchmarks, including 61.0\% accuracy on GSMClaims. Overall, explicit, supervised reasoning enables compact verifiers that are competitive while remaining resource-efficient and interpretable.

Paper Structure

This paper contains 29 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Claim verification and benchmarks.
  4. Reasoning-augmented verification.
  5. Lightweight and specialized verifiers.
  6. Supervising reasoning.
  7. Problem Formulation
  8. Dataset and Model Development
  9. LLMAggreFact-Think Dataset Construction
  10. ThinknCheck-1B Model Training
  11. Ablation Model Variants: Base model, Non-thinking, CoT, and GRPO
  12. GSMClaims Dataset for Arithmetic Reasoning
  13. ThinknCheck-Science: Specializing for Complex Claims
  14. Experiments
  15. Evaluation Metrics
  16. ...and 14 more sections

Figures (3)

  • Figure 1: A sample from the LLMAggreFact-Think dataset, which also illustrates our formulation of the claim verification task: Given a pair of claim and document, our goal is to produce cogent reasoning in addition to the verification label. The [...] represents parts of the reasoning tokens that we elided to accommodate the example in this figure.
  • Figure 2: Reasoning length vs. balanced accuracy (LLMAggreFact). ThinknCheck-1B outputs are grouped into deciles by the token length of the <REASONING> span (Gemma tokenizer). BAcc peaks for mid-length rationales and drops for very short and very long chains. Short chains show recall $>$ precision; very long chains show the opposite trend.
  • Figure 3: Distribution of error types on (a) LLMAggreFact, (b) SciFact, and (c) GSMClaims. Error profiles vary dramatically by domain: general claims are dominated by lexical overlap and aggregation failures; scientific claims by overcautiousness; mathematical claims by arithmetic reasoning errors.