DéjàQ: Open-Ended Evolution of Diverse, Learnable and Verifiable Problems

TL;DR

DéjàQ tackles the problem of static data limitations in reasoning-model training by evolving a diverse, verifiable, and learnable set of synthetic math problems in tandem with model optimization. It combines a MAP-Elites quality-diversity archive with RLVR-based post-training and introduces two LLM-guided mutation strategies to reshape problems, ensuring verifiability and ongoing learning signal. Empirical results show that learnability-driven evolution, especially when using the full set of mutations, yields superior in- and out-of-distribution performance and improved robustness to hard instances, while maintaining reasonable resource usage. The work highlights the potential of continually adapting training data to the model's current capabilities and points to broad applicability beyond mathematics, with plans to open-source code and datasets.

Abstract

Recent advances in reasoning models have yielded impressive results in mathematics and coding. However, most approaches rely on static datasets, which have been suggested to encourage memorisation and limit generalisation. We introduce DéjàQ, a framework that departs from this paradigm by jointly evolving a diverse set of synthetic mathematical problems alongside model training. This evolutionary process adapts to the model's ability throughout training, optimising problems for learnability. We propose two LLM-driven mutation strategies in which the model itself mutates the training data, either by altering contextual details or by directly modifying problem structure. We find that the model can generate novel and meaningful problems, and that these LLM-driven mutations improve RL training. We analyse key aspects of DéjàQ, including the validity of generated problems and computational overhead. Our results underscore the potential of dynamically evolving training data to enhance mathematical reasoning and indicate broader applicability, which we will support by open-sourcing our code.

Figures (4)

• Figure 1: Overview of DéjàQ. We maintain an archive of problem-answer pairs, organised by the setting each question applies to. Training data for RLVR is sampled from this archive, which is continuously updated through various mutators. The setting mutator changes the setting (e.g., from Personal Life to Events), the distractor mutator introduces irrelevant information, and the symbolic mutator alters the underlying mathematical structure. Each problem is scored by its learnability and retained or replaced accordingly.
• Figure 2: Example LLM-guided mutations of a fog coverage problem under the operators used in DéjàQ. Shown are real generations produced by the 7B base model and obtained using the same prompts as applied during training.
• Figure 3: Mean accuracy under conditional value at risk (CVaR) across the six evaluation datasets. The x-axis denotes the risk parameter $\alpha$ (log scale), the y-axis shows mean accuracy, and shaded regions indicate $95\%$ confidence intervals.
• Figure 4: Estimated probabilities with 95% confidence intervals. The left column shows $P(\text{invalid} \mid x \ge \tau)$, i.e., the probability that a question is invalid given that its learnability or depth exceeds a threshold $\tau$. The right column shows the reverse conditional, $P(x \ge \tau \mid \text{invalid})$.