UltraLogic: Enhancing LLM Reasoning through Large-Scale Data Synthesis and Bipolar Float Reward

TL;DR

The paper tackles the data scarcity and reward-signal challenges in training LLMs for general reasoning by introducing UltraLogic, a Code-based Solving framework that auto-generates diverse, difficulty-calibrated reasoning data through seed tasks and programmatic expansion. It decouples logical core from natural language, employs a three-component data framework, and uses a Difficulty Control Module to align task difficulty with model capacity. A key contribution is the Bipolar Float Reward (BFR), which provides dense graded penalties to break the non-informative binary reward trap and accelerate convergence toward global logical optima. Empirical results show that task diversity and the BFR signal substantially improve training efficiency and final reasoning accuracy, though the approach relies on human annotation for seed validation and uses heuristic reward tuning that warrants further study. Overall, UltraLogic offers a scalable path to generate high-quality reasoning data and demonstrates how graded reward signals can enhance RL-based reasoning, with implications for more robust and data-efficient general reasoning in LLMs.

Abstract

While Large Language Models (LLMs) have demonstrated significant potential in natural language processing , complex general-purpose reasoning requiring multi-step logic, planning, and verification remains a critical bottleneck. Although Reinforcement Learning with Verifiable Rewards (RLVR) has succeeded in specific domains , the field lacks large-scale, high-quality, and difficulty-calibrated data for general reasoning. To address this, we propose UltraLogic, a framework that decouples the logical core of a problem from its natural language expression through a Code-based Solving methodology to automate high-quality data production. The framework comprises hundreds of unique task types and an automated calibration pipeline across ten difficulty levels. Furthermore, to mitigate binary reward sparsity and the Non-negative Reward Trap, we introduce the Bipolar Float Reward (BFR) mechanism, utilizing graded penalties to effectively distinguish perfect responses from those with logical flaws. Our experiments demonstrate that task diversity is the primary driver for reasoning enhancement , and that BFR, combined with a difficulty matching strategy, significantly improves training efficiency, guiding models toward global logical optima.

Figures (6)

• Figure 1: The overall architecture of the UltraLogic Data framework.
• Figure 2: Four scoring methods of the BFR mechanism and illustrative examples. Each task type is matched with the most appropriate scoring method to effectively quantify the correctness of model responses based on its unique characteristics, mapping partially correct answers into graded negative penalties.
• Figure 3: The critic/score/mean metrics of Qwen3-8B during the GRPO process by using different reward mechanisms.
• Figure 5: The critic/score/mean metrics of Qwen3-8B during the GRPO process on the different training sets of varying difficulty levels, including Easy, Medium, and Hard.
• Figure 6: The critic/score/mean metrics of Qwen3-14B during the GRPO process on the different training sets of varying difficulty levels, including Easy, Medium, and Hard.