When Small Models Are Right for Wrong Reasons: Process Verification for Trustworthy Agents

TL;DR

Small language-model agents (7–9B) can produce correct answers with fundamentally flawed reasoning in a large fraction of cases, exposing a reliability gap not captured by final accuracy. The authors introduce the Reasoning Integrity Score (RIS) and a distilled verifier to enable process-based verification, and show that Retrieval-Augmented Generation (RAG) robustly improves reasoning integrity while self-critique and verification prompts often degrade it in these models. Across 10,734 traces from three models and three tasks, RIS reveals a 50–69% prevalence of hidden errors in correct outputs, with RAG reducing calculation errors and a fast verifier enabling real-time trust alerts. These findings argue for shifting evaluation and safety practices from outcome-only metrics to process-based auditing to build trustworthy autonomous agents in practical deployments.

Abstract

Deploying small language models (7-9B parameters) as autonomous agents requires trust in their reasoning, not just their outputs. We reveal a critical reliability crisis: 50-69\% of correct answers from these models contain fundamentally flawed reasoning -- a ``Right-for-Wrong-Reasons'' phenomenon invisible to standard accuracy metrics. Through analysis of 10,734 reasoning traces across three models and diverse tasks, we introduce the Reasoning Integrity Score (RIS), a process-based metric validated with substantial inter-rater agreement ($κ=0.657$). Conventional practices are challenged by our findings: while retrieval-augmented generation (RAG) significantly improves reasoning integrity (Cohen's $d=0.23$--$0.93$), meta-cognitive interventions like self-critique often harm performance ($d=-0.14$ to $-0.33$) in small models on the evaluated tasks. Mechanistic analysis reveals RAG succeeds by grounding calculations in external evidence, reducing errors by 7.6\%, while meta-cognition amplifies confusion without sufficient model capacity. To enable deployment, verification capabilities are distilled into a neural classifier achieving 0.86 F1-score with 100$\times$ speedup. These results underscore the necessity of process-based verification for trustworthy agents: accuracy alone is dangerously insufficient when models can be right for entirely wrong reasons.

Figures (1)

• Figure 1: Cohen's $d$ effect sizes for interventions. Red indicates improved reasoning integrity (positive $d$), while Blue indicates harm (negative $d$).