Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement

Fangyuan Li, Pengfei Li, Shijie Wang, Junqi Gao, Jianxing Liu, Biqing Qi, Yuqiang Li

Abstract

Recent progress in reinforcement learning with verifiable rewards (RLVR) offers a practical path to self-improvement of language models, but existing methods face a key trade-off: endogenous self-play can drift over iterations, while corpus-grounded approaches rely on curated data environments. We present \textbf{WIST}, a \textbf{W}eb-grounded \textbf{I}terative \textbf{S}elf-play \textbf{T}ree framework for domain-targeted reasoning improvement that learns directly from the open web without requiring any pre-arranged domain corpus. WIST incrementally expands a domain tree for exploration, and retrieves and cleans path-consistent web corpus to construct a controllable training environment. It then performs Challenger--Solver self-play with verifiable rewards, and feeds learnability signals back to update node posteriors and guide subsequent exploration through an adaptive curriculum. Across four backbones, WIST consistently improves over the base models and typically outperforms both purely endogenous self-evolution and corpus-grounded self-play baselines, with the Overall gains reaching \textbf{+9.8} (\textit{Qwen3-4B-Base}) and \textbf{+9.7} (\textit{OctoThinker-8B}). WIST is also domain-steerable, improving \textit{Qwen3-8B-Base} by \textbf{+14.79} in medicine and \textit{Qwen3-4B-Base} by \textbf{+5.28} on PhyBench. Ablations further confirm the importance of WIST's key components for stable open-web learning. Our Code is available at https://github.com/lfy-123/WIST.

WIST: Web-Grounded Iterative Self-Play Tree for Domain-Targeted Reasoning Improvement

Abstract

Recent progress in reinforcement learning with verifiable rewards (RLVR) offers a practical path to self-improvement of language models, but existing methods face a key trade-off: endogenous self-play can drift over iterations, while corpus-grounded approaches rely on curated data environments. We present \textbf{WIST}, a \textbf{W}eb-grounded \textbf{I}terative \textbf{S}elf-play \textbf{T}ree framework for domain-targeted reasoning improvement that learns directly from the open web without requiring any pre-arranged domain corpus. WIST incrementally expands a domain tree for exploration, and retrieves and cleans path-consistent web corpus to construct a controllable training environment. It then performs Challenger--Solver self-play with verifiable rewards, and feeds learnability signals back to update node posteriors and guide subsequent exploration through an adaptive curriculum. Across four backbones, WIST consistently improves over the base models and typically outperforms both purely endogenous self-evolution and corpus-grounded self-play baselines, with the Overall gains reaching \textbf{+9.8} (\textit{Qwen3-4B-Base}) and \textbf{+9.7} (\textit{OctoThinker-8B}). WIST is also domain-steerable, improving \textit{Qwen3-8B-Base} by \textbf{+14.79} in medicine and \textit{Qwen3-4B-Base} by \textbf{+5.28} on PhyBench. Ablations further confirm the importance of WIST's key components for stable open-web learning. Our Code is available at https://github.com/lfy-123/WIST.
Paper Structure (56 sections, 17 equations, 5 figures, 8 tables)

This paper contains 56 sections, 17 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Reinforcement Learning with Verifiable Rewards
  4. Group Relative Policy Optimization Done Right
  5. Methodology
  6. Overview
  7. Dynamic Domain Tree Construction
  8. Hierarchical domain tree $\mathcal{T}$.
  9. Node-wise learnability posterior and path sampling.
  10. Unk-triggered expansion.
  11. Web-backed validation for non-leaf nodes.
  12. Open-Web Retrieval, Filtering, and Corpus Construction
  13. Web-grounded Two Roles Self-Play Training
  14. Challenger: QA generation with verifiability filtering.
  15. Solver: answer generation and solvability estimation.
  16. ...and 41 more sections

Figures (5)

  • Figure 1: Comparison of R-Zero, SPICE, and our WIST.
  • Figure 2: Overview of our proposed WIST, a web-grounded iterative self-play Tree framework for domain-targeted reasoning improvement.
  • Figure 3: Training performance of our method WIST in the medical domain using three Qwen3 models of different sizes.
  • Figure 4: Ablation results of WIST, including domain transfer, hyperparameter sensitivity, tree scaling, and exploration strategy.
  • Figure 5: Example of a partial mathematics domain label tree generated by WIST. Each non-leaf level contains an unk child that indicates a possible expansion point.