Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineering

Chak Tou Leong, Dingwei Chen, Heming Xia, Qingyu Yin, Sunbowen Lee, Jian Wang, Wenjie Li

TL;DR

The paper introduces RELIEF, a belief-engineering framework that shapes Large Reasoning Models (LRMs) by aligning their internal reasoning beliefs with a target blueprint, bypassing explicit reasoning-trace supervision. It first demonstrates that LRMs harbor latent, probe-accessible beliefs about their reasoning traits via logit probing, then constructs a Target Belief Blueprint and synthesizes self-reflective QA data to internalize the desired traits. After internalization through supervised fine-tuning on belief-affirming QA, RELIEF biases subsequent reasoning towards improved efficiency and faithfulness. Across efficiency and faithfulness tasks, RELIEF matches or surpasses trace-based supervised and preference-based baselines while requiring substantially fewer training tokens, illustrating a scalable, data-efficient path to shaping reasoning behavior through internal beliefs.

Abstract

Large reasoning models (LRMs) have achieved remarkable success in complex problem-solving, yet they often suffer from computational redundancy or reasoning unfaithfulness. Current methods for shaping LRM behavior typically rely on reinforcement learning or fine-tuning with gold-standard reasoning traces, a paradigm that is both computationally expensive and difficult to scale. In this paper, we reveal that LRMs possess latent \textit{reasoning beliefs} that internally track their own reasoning traits, which can be captured through simple logit probing. Building upon this insight, we propose Reasoning Belief Engineering (RELIEF), a simple yet effective framework that shapes LRM behavior by aligning the model's self-concept with a target belief blueprint. Crucially, RELIEF completely bypasses the need for reasoning-trace supervision. It internalizes desired traits by fine-tuning on synthesized, self-reflective question-answering pairs that affirm the target belief. Extensive experiments on efficiency and faithfulness tasks demonstrate that RELIEF matches or outperforms behavior-supervised and preference-based baselines while requiring lower training costs. Further analysis validates that shifting a model's reasoning belief effectively shapes its actual behavior.

Finding RELIEF: Shaping Reasoning Behavior without Reasoning Supervision via Belief Engineering

TL;DR

The paper introduces RELIEF, a belief-engineering framework that shapes Large Reasoning Models (LRMs) by aligning their internal reasoning beliefs with a target blueprint, bypassing explicit reasoning-trace supervision. It first demonstrates that LRMs harbor latent, probe-accessible beliefs about their reasoning traits via logit probing, then constructs a Target Belief Blueprint and synthesizes self-reflective QA data to internalize the desired traits. After internalization through supervised fine-tuning on belief-affirming QA, RELIEF biases subsequent reasoning towards improved efficiency and faithfulness. Across efficiency and faithfulness tasks, RELIEF matches or surpasses trace-based supervised and preference-based baselines while requiring substantially fewer training tokens, illustrating a scalable, data-efficient path to shaping reasoning behavior through internal beliefs.

Abstract

Large reasoning models (LRMs) have achieved remarkable success in complex problem-solving, yet they often suffer from computational redundancy or reasoning unfaithfulness. Current methods for shaping LRM behavior typically rely on reinforcement learning or fine-tuning with gold-standard reasoning traces, a paradigm that is both computationally expensive and difficult to scale. In this paper, we reveal that LRMs possess latent \textit{reasoning beliefs} that internally track their own reasoning traits, which can be captured through simple logit probing. Building upon this insight, we propose Reasoning Belief Engineering (RELIEF), a simple yet effective framework that shapes LRM behavior by aligning the model's self-concept with a target belief blueprint. Crucially, RELIEF completely bypasses the need for reasoning-trace supervision. It internalizes desired traits by fine-tuning on synthesized, self-reflective question-answering pairs that affirm the target belief. Extensive experiments on efficiency and faithfulness tasks demonstrate that RELIEF matches or outperforms behavior-supervised and preference-based baselines while requiring lower training costs. Further analysis validates that shifting a model's reasoning belief effectively shapes its actual behavior.
Paper Structure (52 sections, 5 equations, 19 figures, 4 tables)

This paper contains 52 sections, 5 equations, 19 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivating Analysis: Latent Belief of LRM's Reasoning Behavior
  3. Probing Setup.
  4. Results.
  5. Takeaway.
  6. Methodology
  7. Target Belief Construction.
  8. Belief-Reflecting QA Synthesis.
  9. Target Belief Internalization.
  10. Experiments
  11. Shaping Reasoning Efficiency
  12. Datasets and Metrics.
  13. Baselines.
  14. Results.
  15. Training Efficiency.
  16. ...and 37 more sections

Figures (19)

  • Figure 1: Probing LRM latent beliefs. Logit differences in efficiency (upper; Qwen3-8B) and faithfulness (bottom; R1-Qwen-7B) correlate significantly with external metrics.
  • Figure 2: Overview of the proposed RELIEF framework for shaping LRMs' reasoning behaviors.
  • Figure 3: Performance vs. Reasoning Length.RELIEF achieves high accuracy-efficiency trade-off using only 0.4M training tokens, significantly fewer than methods relying on reasoning traces ($>$3.9M).
  • Figure 4: Faithfulness shaping results on MMLU-Redux and GPQA-Diamond under three hint types. RELIEF consistently improves faithfulness scores over the original models and prompting baselines, matching or surpassing SimPO without using reasoning traces. $\dagger$ indicates low confidence.
  • Figure 5: Control analysis on Efficiency.RELIEF (Efficient) achieves lossless compression, improving accuracy where the SFT baseline (None) fails. The Inefficient target further confirms effective bidirectional shaping on R1-Qwen-7B.
  • ...and 14 more figures