Reasoning over mathematical objects: on-policy reward modeling and test time aggregation

Abstract

The ability to precisely derive mathematical objects is a core requirement for downstream STEM applications, including mathematics, physics, and chemistry, where reasoning must culminate in formally structured expressions. Yet, current LM evaluations of mathematical and scientific reasoning rely heavily on simplified answer formats such as numerical values or multiple choice options due to the convenience of automated assessment. In this paper we provide three contributions for improving reasoning over mathematical objects: (i) we build and release training data and benchmarks for deriving mathematical objects, the Principia suite; (ii) we provide training recipes with strong LLM-judges and verifiers, where we show that on-policy judge training boosts performance; (iii) we show how on-policy training can also be used to scale test-time compute via aggregation. We find that strong LMs such as Qwen3-235B and o3 struggle on Principia, while our training recipes can bring significant improvements over different LLM backbones, while simultaneously improving results on existing numerical and MCQA tasks, demonstrating cross-format generalization of reasoning abilities.

Figures (31)

• Figure 1: RL training on the Principia Collection, which requires mathematical-object outputs, improves an LM’s reasoning capability. (a) On our new challenging benchmark, PrincipiaBench, which requires deriving mathematical objects, Qwen3-4B-Base, Qwen2.5-7B-Base, and OctoThinker-8B-Long-Base trained on Principia Collection yield average gains of +18.23%, +10.23%, and +15.16%, respectively. (b) Training on the Principia Collection also improves performance on numerical (AIME-2025) and MCQA (SuperGPQA) benchmarks, demonstrating cross-format reasoning gains. (c) Using a strong model-based verifier (GPT-OSS-120B) and excluding MCQA data provides the best performance.
• Figure 2: LMs struggle to solve problems that require mathematical objects as answers. For example, Qwen3-235B can solve this problem from SuperGPQA du2025supergpqa in an MCQ setting by using the options as an anchor to perform backward chaining. However, when removing the options, the model starts making false assumptions (i.e., that all 1-eigenspaces share a common fixed vector and that the representation cannot be fully trivial) and derives the incorrect solution (i.e., concluding $\langle 1_G, \chi \rangle = 1$ instead of allowing the valid case $\langle 1_G, \chi \rangle = 2$). Note that yellow-highlighted comments pinpoint and describe the critical flaws included within the model's chain-of-thought.
• Figure 3: Performance consistently drops when removing options in MCQA benchmarks. In the mathematical and engineering subset of SuperGPQA, among instances where answers are expressed as mathematical objects, LMs show a 10–-20% decrease in performance when options are removed. This suggests that MCQA evaluations tend to overestimate true reasoning ability (i.e., high MCQA scores do not necessarily translate into strong reasoning over mathematical objects).
• Figure 4: Token count distribution comparison between the Principia Collection and other widely used RL post-training datasets. The problem statements and answers of Principia Collection are relatively longer since it requires the derivation of complex mathematical objects and the problem statements based on graduate-level STEM subjects are very detailed.
• Figure 5: Annotation UI for constructing the Principia VerifyBench.