Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QueST: Incentivizing LLMs to Generate Difficult Problems

Hanxu Hu, Xingxing Zhang, Jannis Vamvas, Rico Sennrich, Furu Wei

TL;DR

QueST addresses the bottleneck of lacking large-scale, difficult training data for code reasoning by training a specialized problem generator. It combines difficulty-aware graph sampling and rejection fine-tuning to systematically produce hard coding problems, enabling effective distillation and reinforcement learning for smaller models. Empirical results show QueST-generated data yields strong performance on LiveCodeBench and USACO, approaching or matching much larger models when combined with human data. The work demonstrates a scalable pathway to push the frontiers of code reasoning in LLMs, while acknowledging computational cost and the need for future real-time reward mechanisms.

Abstract

Large Language Models have achieved strong performance on reasoning tasks, solving competition-level coding and math problems. However, their scalability is limited by human-labeled datasets and the lack of large-scale, challenging coding problem training data. Existing competitive coding datasets contain only thousands to tens of thousands of problems. Previous synthetic data generation methods rely on either augmenting existing instruction datasets or selecting challenging problems from human-labeled data. In this paper, we propose QueST, a novel framework which combines difficulty-aware graph sampling and difficulty-aware rejection fine-tuning that directly optimizes specialized generators to create challenging coding problems. Our trained generators demonstrate superior capability compared to even GPT-4o at creating challenging problems that benefit downstream performance. We leverage QueST to generate large-scale synthetic coding problems, which we then use to distill from strong teacher models with long chain-of-thought or to conduct reinforcement learning for smaller models, proving effective in both scenarios. Our distillation experiments demonstrate significant performance gains. Specifically, after fine-tuning Qwen3-8B-base on 100K difficult problems generated by QueST, we surpass the performance of the original Qwen3-8B on LiveCodeBench. With an additional 112K examples (i.e., 28K human-written problems paired with multiple synthetic solutions), our 8B model matches the performance of the much larger DeepSeek-R1-671B. These findings indicate that generating complex problems via QueST offers an effective and scalable approach to advancing the frontiers of competitive coding and reasoning for large language models.

QueST: Incentivizing LLMs to Generate Difficult Problems

TL;DR

QueST addresses the bottleneck of lacking large-scale, difficult training data for code reasoning by training a specialized problem generator. It combines difficulty-aware graph sampling and rejection fine-tuning to systematically produce hard coding problems, enabling effective distillation and reinforcement learning for smaller models. Empirical results show QueST-generated data yields strong performance on LiveCodeBench and USACO, approaching or matching much larger models when combined with human data. The work demonstrates a scalable pathway to push the frontiers of code reasoning in LLMs, while acknowledging computational cost and the need for future real-time reward mechanisms.

Abstract

Large Language Models have achieved strong performance on reasoning tasks, solving competition-level coding and math problems. However, their scalability is limited by human-labeled datasets and the lack of large-scale, challenging coding problem training data. Existing competitive coding datasets contain only thousands to tens of thousands of problems. Previous synthetic data generation methods rely on either augmenting existing instruction datasets or selecting challenging problems from human-labeled data. In this paper, we propose QueST, a novel framework which combines difficulty-aware graph sampling and difficulty-aware rejection fine-tuning that directly optimizes specialized generators to create challenging coding problems. Our trained generators demonstrate superior capability compared to even GPT-4o at creating challenging problems that benefit downstream performance. We leverage QueST to generate large-scale synthetic coding problems, which we then use to distill from strong teacher models with long chain-of-thought or to conduct reinforcement learning for smaller models, proving effective in both scenarios. Our distillation experiments demonstrate significant performance gains. Specifically, after fine-tuning Qwen3-8B-base on 100K difficult problems generated by QueST, we surpass the performance of the original Qwen3-8B on LiveCodeBench. With an additional 112K examples (i.e., 28K human-written problems paired with multiple synthetic solutions), our 8B model matches the performance of the much larger DeepSeek-R1-671B. These findings indicate that generating complex problems via QueST offers an effective and scalable approach to advancing the frontiers of competitive coding and reasoning for large language models.
Paper Structure (23 sections, 10 equations, 7 figures, 7 tables)

This paper contains 23 sections, 10 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. QueST
  3. Preliminary: Problem Generation Through Concept Graph
  4. Concept Extraction
  5. Graph Construction
  6. Problem Generation
  7. Difficulty-aware Rejection Finetuning
  8. Difficulty Estimation
  9. Rejection Fine-tuning
  10. Difficulty-aware Graph Construction
  11. Experiments
  12. Implementation Details
  13. Using Estimated Difficulty For Data Selection
  14. Trained Generator for Distillation
  15. Reinforcement Learning
  16. ...and 8 more sections

Figures (7)

  • Figure 1: Comparisons of Livecodebench scores and model parameters between LLMs trained using various methods. Our model (QueST-8B) achieves a new Pareto optimum.
  • Figure 2: The pipeline of QueST. We first extract concepts based on seed problems, then use difficulty-aware sampling method described in Equation \ref{['eq:edge-weight-diff']} to create prompts for problem generation. We generate 8 problems for each prompt, calculate the difficulty $\delta$ of the generated problem based on Equation \ref{['eq:diff']}, and use the most difficult problem as rejection fine-tuning data to train our generator.
  • Figure 3: Training rewards comparison in the training process of RL under different datasets.
  • Figure 4: Examples of knowledge points where the count is increased (left) or decreased (right) when using difficulty-aware sampling.
  • Figure 5: 1-shot prompt example for problem generation. It is simplified for visualization, in real prompt, we have 8-shot for in-context learning.
  • ...and 2 more figures