Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Questioning Language Models

Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, Deepak Pathak

TL;DR

This work introduces Self-Questioning Language Models (SQLM), an asymmetric self-play framework in which a topic-constrained proposer creates questions and a solver attempts to answer them, with both roles trained through reinforcement learning using internal agreement or unit-test verification as rewards. When the generator-verifier gap is small, majority voting serves as a proxy for correctness; when the gap is large, unit tests provide verification signals. The authors evaluate SQLM on arithmetic, algebra, and coding tasks, showing meaningful improvements without external labeled data and highlighting the potential of self-generated curricula for post-training refinement. The approach reduces reliance on hand-crafted datasets and prompts, though it acknowledges remaining prompting overhead and the risk of reinforcing errors without external grounding. Overall, SQLM demonstrates a promising path toward fully self-supervised self-improvement in language models and invites further automation and validation across domains.

Abstract

Can large language models improve without external data -- by generating their own questions and answers? We hypothesize that a pre-trained language model can improve its reasoning skills given only a single prompt specifying the topic (e.g., algebra word problems) and asking the model to generate its own questions. To do this, we propose Self-Questioning Language Models (SQLM): an asymmetric self-play framework where a proposer is given the topic and generates a question for a solver, who tries to answer it. Both the proposer and solver are trained via reinforcement learning. The proposer receives a reward if the problem is not too easy or too difficult, and the solver receives a reward based on majority voting, a proxy for correctness in the absence of ground-truth answers. For coding, the proposer can instead generate unit tests which are used for verification. We study this asymmetric self-play framework on three benchmarks: three-digit multiplication, algebra problems from the OMEGA benchmark, and programming problems from Codeforces. By continually generating more interesting problems and attempting to solve them, language models can improve on downstream benchmarks without access to any curated training datasets.

Self-Questioning Language Models

TL;DR

This work introduces Self-Questioning Language Models (SQLM), an asymmetric self-play framework in which a topic-constrained proposer creates questions and a solver attempts to answer them, with both roles trained through reinforcement learning using internal agreement or unit-test verification as rewards. When the generator-verifier gap is small, majority voting serves as a proxy for correctness; when the gap is large, unit tests provide verification signals. The authors evaluate SQLM on arithmetic, algebra, and coding tasks, showing meaningful improvements without external labeled data and highlighting the potential of self-generated curricula for post-training refinement. The approach reduces reliance on hand-crafted datasets and prompts, though it acknowledges remaining prompting overhead and the risk of reinforcing errors without external grounding. Overall, SQLM demonstrates a promising path toward fully self-supervised self-improvement in language models and invites further automation and validation across domains.

Abstract

Can large language models improve without external data -- by generating their own questions and answers? We hypothesize that a pre-trained language model can improve its reasoning skills given only a single prompt specifying the topic (e.g., algebra word problems) and asking the model to generate its own questions. To do this, we propose Self-Questioning Language Models (SQLM): an asymmetric self-play framework where a proposer is given the topic and generates a question for a solver, who tries to answer it. Both the proposer and solver are trained via reinforcement learning. The proposer receives a reward if the problem is not too easy or too difficult, and the solver receives a reward based on majority voting, a proxy for correctness in the absence of ground-truth answers. For coding, the proposer can instead generate unit tests which are used for verification. We study this asymmetric self-play framework on three benchmarks: three-digit multiplication, algebra problems from the OMEGA benchmark, and programming problems from Codeforces. By continually generating more interesting problems and attempting to solve them, language models can improve on downstream benchmarks without access to any curated training datasets.

Paper Structure

This paper contains 25 sections, 6 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for Language Models
  4. Unsupervised Rewards for Reasoning
  5. Exploration
  6. Synthetic Data Generation
  7. Preliminaries
  8. Reinforcement Learning for Language Models
  9. Asymmetric Self-Play
  10. Method
  11. Minimax Objective
  12. Reward Functions
  13. Experiments
  14. Setups
  15. Main Results
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Overview of Self-Questioning Language Models. The only input to the system is a single prompt, given to the proposer. The proposer generates a question related to the given topic, and the solver aims to solve the question. The solver's reward is computed by using the majority vote as a proxy for the ground-truth answer. The proposer's reward is computed based on how many of the answers match the majority answer, to encourage problems not to be too easy or too difficult.
  • Figure 2: Comparison of generating all problems at once vs one at a time using the proposer.