Counterfactual Self-Questioning for Stable Policy Optimization in Language Models

TL;DR

Counterfactual Self-Questioning (CSQ) enables a single language model to generate and evaluate counterfactual critiques of its own reasoning without external verifiers or reward models. It uses a base chain-of-thought and internally generated counterfactual trajectories to produce structured learning signals via Group Relative Policy Optimization (GRPO), improving reasoning stability and accuracy. Across GSM8K, MATH, and Minerva-style benchmarks, CSQ yields consistent gains, particularly for small- to mid-sized models, with one or two ego critics providing the best balance between critique diversity and optimization stability. The approach offers a scalable, verifier-free path to self-improvement that internalizes critique into policy updates, reducing reliance on ensembles or external validators.

Abstract

Recent work on language model self-improvement shows that models can refine their own reasoning through reflection, verification, debate, or self-generated rewards. However, most existing approaches rely on external critics, learned reward models, or ensemble sampling, which increases complexity and training instability. We propose Counterfactual Self-Questioning, a framework in which a single language model generates and evaluates counterfactual critiques of its own reasoning. The method produces an initial reasoning trace, formulates targeted questions that challenge potential failure points, and generates alternative reasoning trajectories that expose incorrect assumptions or invalid steps. These counterfactual trajectories provide structured relative feedback that can be directly used for policy optimization without auxiliary models. Experiments on multiple mathematical reasoning benchmarks show that counterfactual self-questioning improves accuracy and training stability, particularly for smaller models, enabling scalable self-improvement using internally generated supervision alone.

Figures (1)

• Figure 1: Overview of Counterfactual Self-Questioning (CSQ). A base policy generates an initial reasoning trajectory. Counterfactual self-questioning produces alternative trajectories that expose failure modes. These trajectories are used as relative feedback for policy optimization via GRPO.