Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text

TL;DR

GooseReason introduces VeraRL, a scalable data-synthesis pipeline that converts reasoning-rich, unverifiable internet text into verifiable RLVR tasks using a fill-in-the-middle MCQ format. By masking the crux of reasoning and generating diverse distractors, it creates an effectively endless stream of RLVR data, enabling continual RL scaling beyond traditional data saturation. The approach yields state-of-the-art performance across math, coding, STEM, and cybersecurity benchmarks and demonstrates data-efficient scaling under compute constraints. The work emphasizes the practical potential of leveraging vast, reasoning-rich text for scalable RL and releases VeraRL to spur further research in verifiable reasoning tasks.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a cornerstone for unlocking complex reasoning in Large Language Models (LLMs). Yet, scaling up RL is bottlenecked by limited existing verifiable data, where improvements increasingly saturate over prolonged training. To overcome this, we propose Golden Goose, a simple trick to synthesize unlimited RLVR tasks from unverifiable internet text by constructing a multiple-choice question-answering version of the fill-in-the-middle task. Given a source text, we prompt an LLM to identify and mask key reasoning steps, then generate a set of diverse, plausible distractors. This enables us to leverage reasoning-rich unverifiable corpora typically excluded from prior RLVR data construction (e.g., science textbooks) to synthesize GooseReason-0.7M, a large-scale RLVR dataset with over 0.7 million tasks spanning mathematics, programming, and general scientific domains. Empirically, GooseReason effectively revives models saturated on existing RLVR data, yielding robust, sustained gains under continuous RL and achieving new state-of-the-art results for 1.5B and 4B-Instruct models across 15 diverse benchmarks. Finally, we deploy Golden Goose in a real-world setting, synthesizing RLVR tasks from raw FineWeb scrapes for the cybersecurity domain, where no prior RLVR data exists. Training Qwen3-4B-Instruct on the resulting data GooseReason-Cyber sets a new state-of-the-art in cybersecurity, surpassing a 7B domain-specialized model with extensive domain-specific pre-training and post-training. This highlights the potential of automatically scaling up RLVR data by exploiting abundant, reasoning-rich, unverifiable internet text.

Figures (14)

• Figure 1: The olden oose pipeline. We synthesize RLVR tasks from unverifiable text by constructing a MCQ version of the fill-in-the-middle task. Given a source text, we prompt an LLM to first identify a contiguous span of crucial reasoning steps and replace it with a [MASK], treating the removed content as the ground-truth answer, and then generate a set of diverse distractors that are plausible and similar to the masked span, yet incorrect. For noisy data sources (e.g., web scrapes), we prompt the LLM to first extract an educationally valuable passage and then construct the MCQ based on it. We further apply difficulty-based filtering to remove easy problems.
• Figure 2: Comparison of continued RL training on Qwen-4B-Instruct after data saturation using the original ProRL data versus adding GooseReason-0.7M. The former exhibits performance plateaus or regression, while the latter yields robust, continuous gains.
• Figure 3: Comparison between GooseReason-0.7M and existing RLVR datasets used in ProRL liu2025prorl in terms of total examples and effective examples, measured relative to ProRL-1.5B-v2. We define an example as effective if it has both successful and failed model rollouts, yielding meaningful learning signal for RL. Notably, we increase the number of effective examples in math, code, and STEM by over 450,000, which is a 13$\times$ increase over the total effective examples in the ProRL dataset.
• Figure 4: Accuracy distribution of ProRL-1.5B-v2, calculated as the success rate over 16 rollouts per task, on GooseReason-Math across different task formulations. Notably, with 9-choice MCQ format, the majority of problems fall into a medium-difficulty regime (exhibiting both successful and failed model rollouts), providing the most effective signals for RL training.
• Figure 5: Comparison of continued RL training on ProRL-1.5B-v2 using the original ProRL data, adding GooseReason-0.7M, or using RLVEZeng2025RLVESU. Continuing with ProRL data yields marginal gains, adding GooseReason-0.7M produces robust, continuous improvements, while RLVE is highly effective in math but less so in STEM and coding.