Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution

Nithin Sivakumaran, Shoubin Yu, Hyunji Lee, Yue Zhang, Ali Payani, Mohit Bansal, Elias Stengel-Eskin

TL;DR

REMUL consistently and substantially improves three measures of faithfulness -- hint attribution, early answering area over the curve (AOC), and mistake injection AOC -- while also improving accuracy and is associated with shorter and more direct CoTs.

Abstract

Chain-of-thought (CoT) reasoning sometimes fails to faithfully reflect the true computation of a large language model (LLM), hampering its utility in explaining how LLMs arrive at their answers. Moreover, optimizing for faithfulness and interpretability in reasoning often degrades task performance. To address this tradeoff and improve CoT faithfulness, we propose Reasoning Execution by Multiple Listeners (REMUL), a multi-party reinforcement learning approach. REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful. A speaker model generates a reasoning trace, which is truncated and passed to a pool of listener models who "execute" the trace, continuing the trace to an answer. Speakers are rewarded for producing reasoning that is clear to listeners, with additional correctness regularization via masked supervised finetuning to counter the tradeoff between faithfulness and performance. On multiple reasoning benchmarks (BIG-Bench Extra Hard, MuSR, ZebraLogicBench, and FOLIO), REMUL consistently and substantially improves three measures of faithfulness -- hint attribution, early answering area over the curve (AOC), and mistake injection AOC -- while also improving accuracy. Our analysis finds that these gains are robust across training domains, translate to legibility gains, and are associated with shorter and more direct CoTs.

Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution

TL;DR

REMUL consistently and substantially improves three measures of faithfulness -- hint attribution, early answering area over the curve (AOC), and mistake injection AOC -- while also improving accuracy and is associated with shorter and more direct CoTs.

Abstract

Chain-of-thought (CoT) reasoning sometimes fails to faithfully reflect the true computation of a large language model (LLM), hampering its utility in explaining how LLMs arrive at their answers. Moreover, optimizing for faithfulness and interpretability in reasoning often degrades task performance. To address this tradeoff and improve CoT faithfulness, we propose Reasoning Execution by Multiple Listeners (REMUL), a multi-party reinforcement learning approach. REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful. A speaker model generates a reasoning trace, which is truncated and passed to a pool of listener models who "execute" the trace, continuing the trace to an answer. Speakers are rewarded for producing reasoning that is clear to listeners, with additional correctness regularization via masked supervised finetuning to counter the tradeoff between faithfulness and performance. On multiple reasoning benchmarks (BIG-Bench Extra Hard, MuSR, ZebraLogicBench, and FOLIO), REMUL consistently and substantially improves three measures of faithfulness -- hint attribution, early answering area over the curve (AOC), and mistake injection AOC -- while also improving accuracy. Our analysis finds that these gains are robust across training domains, translate to legibility gains, and are associated with shorter and more direct CoTs.
Paper Structure (42 sections, 3 equations, 4 figures, 12 tables)

This paper contains 42 sections, 3 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Setup
  4. Cross-model Soft Reasoning Execution
  5. Incorporating Correctness
  6. Balanced Rewards.
  7. Correctness Supervised Finetuning.
  8. Training Procedure
  9. Experiments and Results
  10. Experimental Setup
  11. Training data.
  12. Evaluation datasets.
  13. Implementation details.
  14. Metrics
  15. AOC Metrics.
  16. ...and 27 more sections

Figures (4)

  • Figure 1: Existing reasoning models are prone to producing reasoning that is unfaithful and hard to follow. Thus, when given a reasoning chain prefix (blue) from a base speaker model (A), listener models are often unable to reproduce the speaker's final decision, i.e., their soft execution suffix (white) diverges. REMuL rewards speakers for inducing consistent reasoning in the listener pool. With REMuL, the speaker's reasoning is more faithful and easier for the listeners to follow consistently, with all listeners reaching the same conclusion as the speaker. Note that reasoning here is elided, and the shown prefix and truncated suffix are not by necessity the same length.
  • Figure 2: REMuL consists of two components: (Top) A speaker-listener reasoning execution reward, where listeners execute reasoning prefixes from a speaker, who is rewarded for listener consensus. The speaker's final answer is only used for reward computation and not seen by the listeners. (Bottom) A masked supervised finetuning step to maintain correctness via a LoRA adapter, with loss computed only on answer tokens.
  • Figure 3: Qwen3-14B reward curves during training for a model trained with a balanced, joint reward (Faithfulness + Correctness) as compared to each feature trained separately.
  • Figure 4: All prompt templates used in experiments.