Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Smooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Model

Siwen Jiao, Tianxiong Lv, Kangan Qian, Chenxu Zhao, Xiuyuan Zhu, Tianlun Li, Xiaolong Cheng, Jinyu Li, Zhihao Liao, Yang Cai

TL;DR

This work tackles the challenge of precise numerical prediction in 3D scene understanding with vision-language models, where standard RL is hindered by reward sparsity and gradient instability. It introduces Smooth Numerical Reward Activation (SNRA), a dynamic sigmoid-based reward transform, and Absolute-Preserving GRPO (AP-GRPO), which multiplies relative advantages by an absolute accuracy term to retain boundary information. A Dynamic Sharpness Scheduling scheme guides the optimization from exploration to fine-grained precision, and Numerical3D-50k provides a compact, verifiable 3D task dataset for training. Empirically, AP-GRPO with SNRA achieves competitive results to large-scale supervised baselines using only 50k samples, highlighting substantial data efficiency and the viability of physics-grounded RL for 3D spatial reasoning without extra architectural changes.

Abstract

Vision-Language Models (VLMs) face a critical bottleneck in achieving precise numerical prediction for 3D scene understanding. Traditional reinforcement learning (RL) approaches, primarily based on relative ranking, often suffer from severe reward sparsity and gradient instability, failing to effectively exploit the verifiable signals provided by 3D physical constraints. Notably, in standard GRPO frameworks, relative normalization causes "near-miss" samples (characterized by small but non-zero errors) to suffer from advantage collapse. This leads to a severe data utilization bottleneck where valuable boundary samples are discarded during optimization. To address this, we introduce the Smooth Numerical Reward Activation (SNRA) operator and the Absolute-Preserving GRPO (AP-GRPO) framework. SNRA employs a dynamically parameterized Sigmoid function to transform raw feedback into a dense, continuous reward continuum. Concurrently, AP-GRPO integrates absolute scalar gradients to mitigate the numerical information loss inherent in conventional relative-ranking mechanisms. By leveraging this approach, we constructed Numerical3D-50k, a dataset comprising 50,000 verifiable 3D subtasks. Empirical results indicate that AP-GRPO achieves performance parity with large-scale supervised methods while maintaining higher data efficiency, effectively activating latent 3D reasoning in VLMs without requiring architectural modifications.

Smooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Model

TL;DR

This work tackles the challenge of precise numerical prediction in 3D scene understanding with vision-language models, where standard RL is hindered by reward sparsity and gradient instability. It introduces Smooth Numerical Reward Activation (SNRA), a dynamic sigmoid-based reward transform, and Absolute-Preserving GRPO (AP-GRPO), which multiplies relative advantages by an absolute accuracy term to retain boundary information. A Dynamic Sharpness Scheduling scheme guides the optimization from exploration to fine-grained precision, and Numerical3D-50k provides a compact, verifiable 3D task dataset for training. Empirically, AP-GRPO with SNRA achieves competitive results to large-scale supervised baselines using only 50k samples, highlighting substantial data efficiency and the viability of physics-grounded RL for 3D spatial reasoning without extra architectural changes.

Abstract

Vision-Language Models (VLMs) face a critical bottleneck in achieving precise numerical prediction for 3D scene understanding. Traditional reinforcement learning (RL) approaches, primarily based on relative ranking, often suffer from severe reward sparsity and gradient instability, failing to effectively exploit the verifiable signals provided by 3D physical constraints. Notably, in standard GRPO frameworks, relative normalization causes "near-miss" samples (characterized by small but non-zero errors) to suffer from advantage collapse. This leads to a severe data utilization bottleneck where valuable boundary samples are discarded during optimization. To address this, we introduce the Smooth Numerical Reward Activation (SNRA) operator and the Absolute-Preserving GRPO (AP-GRPO) framework. SNRA employs a dynamically parameterized Sigmoid function to transform raw feedback into a dense, continuous reward continuum. Concurrently, AP-GRPO integrates absolute scalar gradients to mitigate the numerical information loss inherent in conventional relative-ranking mechanisms. By leveraging this approach, we constructed Numerical3D-50k, a dataset comprising 50,000 verifiable 3D subtasks. Empirical results indicate that AP-GRPO achieves performance parity with large-scale supervised methods while maintaining higher data efficiency, effectively activating latent 3D reasoning in VLMs without requiring architectural modifications.
Paper Structure (28 sections, 6 theorems, 51 equations, 4 figures, 8 tables)

This paper contains 28 sections, 6 theorems, 51 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Visual Spatial Understanding and Reasoning
  4. VLM Reinforcement Learning with Verifiable Rewards
  5. Method
  6. 3D Perceptual Subtasks and Numerical3D-50k Dataset
  7. Smooth Numerical Reward Activation Operator (SNRA)
  8. Dynamic Sharpness Scheduling
  9. Absolute-Preserving GRPO (AP-GRPO)
  10. Experiments
  11. Experimental Setup
  12. Main Results
  13. Roadmap of Reward and Architecture Synergy
  14. Ablation Studies
  15. Conclusion
  16. ...and 13 more sections

Key Result

Proposition 1.1

Let $\mathcal{G}$ be a GRPO group, and for $\tau \in \mathcal{G}$ define Assume that reward $r(\tau)$ is monotone in the original advantage, i.e. Then for any $\tau \in \mathcal{G}$: In particular, AP-GRPO guarantees that trajectories with positive relative advantages maintain their local ranking, ensuring that the best-performing samples within a group receive the strongest positive reinforcem

Figures (4)

  • Figure 1: Standard GRPO vs. AP-GRPO(Ours). Standard GRPO (left) assigns binary rewards, yielding zero advantages for near-correct responses and discarding all samples, resulting in no policy update. In contrast, AP-GRPO (right) preserves absolute precision through SNRA smoothing, producing dense rewards and non-zero advantages modulated by the absolute term, thereby enabling effective policy updates and high sample utilization for precise numerical perception.
  • Figure 2: Derivation and mechanism of the SNRA operator. The roadmap depicts the transformation from a standard sigmoid $\sigma(x)$ (Stage 1) to the final reward mapping (Stage 4) via symmetry inversion and amplitude scaling. This formulation anchors the operator at $f(0)=1$ for perfect predictions. The sharpness $k$ acts as a dynamic soft-threshold: lower $k$ provides smooth gradients to encourage exploration, while higher $k$ enforces strict numerical compliance for precision convergence.
  • Figure 3: Overview of the AP-GRPO framework. The method processes continuous (e.g., grounding, depth, size, distance) and discrete numerical tasks (e.g., direction, order, counting, position) via a policy model that generates reasoning and answers. Raw rewards are smoothed by SNRA into dense $R_i'$, then used to compute group-relative advantages modulated by absolute preservation $(R_i')^\alpha$, yielding $A_i'$. Policy updates employ a clipped objective with KL regularization for stable, precise 3D perception.
  • Figure 4: Component-level ablation study. The synergy of SNRA and Dynamic Sharpness allows AP-GRPO to surpass standard GRPO variants, validating the necessity of geometric priors and reward landscape refinement for precise 3D reasoning.

Theorems & Definitions (12)

  • Proposition 1.1: Sign preservation and positive ordering
  • proof
  • Theorem 1.2
  • proof
  • Theorem 1.3
  • proof
  • Lemma 1.4: Gradient Extremum Localization
  • proof
  • Theorem 1.5: Global Search via Low Sharpness
  • proof
  • ...and 2 more