SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization

TL;DR

SSL reframes reinforcement learning reward signals from binary success/failure to tiered, proximity-aware guidance that concentrates learning in the 'sweet-spot' region of the solution space. The method defines $R_{SSL}(\tau)=C(\tau)+\alpha\widehat{S}(\tau)$ with trajectory proximity $S(\tau)$ aggregated from step proximities and discretized into zones, then optimizes with standard RLVR. Theoretical results show SSL preserves optimal solution ordering and improves gradient signal-to-noise ratio, yielding faster, more stable learning. Empirically, SSL delivers consistent gains across 12 benchmarks in GUI perception, short/long-term planning, and complex reasoning, achieving up to 2.5× data efficiency and cross-task transferability. Overall, SSL provides a general, interpretable principle for differentiating guidance in agentic optimization, with broad practical impact on sample efficiency and robustness.

Abstract

Reinforcement learning with verifiable rewards has emerged as a powerful paradigm for training intelligent agents. However, existing methods typically employ binary rewards that fail to capture quality differences among trajectories achieving identical outcomes, thereby overlooking potential diversity within the solution space. Inspired by the ``sweet spot'' concept in tennis-the racket's core region that produces optimal hitting effects, we introduce \textbf{S}weet \textbf{S}pot \textbf{L}earning (\textbf{SSL}), a novel framework that provides differentiated guidance for agent optimization. SSL follows a simple yet effective principle: progressively amplified, tiered rewards guide policies toward the sweet-spot region of the solution space. This principle naturally adapts across diverse tasks: visual perception tasks leverage distance-tiered modeling to reward proximity, while complex reasoning tasks reward incremental progress toward promising solutions. We theoretically demonstrate that SSL preserves optimal solution ordering and enhances the gradient signal-to-noise ratio, thereby fostering more directed optimization. Extensive experiments across GUI perception, short/long-term planning, and complex reasoning tasks show consistent improvements over strong baselines on 12 benchmarks, achieving up to 2.5X sample efficiency gains and effective cross-task transferability. Our work establishes SSL as a general principle for training capable and robust agents.

Figures (10)

• Figure 1: Performance comparison with Binary (RL-B.) and Continuous (RL-C.) Reward RL. SSL (our method) exhibits remarkable improvements across diverse tasks, including long-/short-term planning and complex reasoning.
• Figure 2: Sweet Spot Learning Overview. We visualize sweet-spot zones and their instantiation across diverse tasks .
• Figure 3: Flowchart of SSL. SSL computes step proximities, aggregates them into trajectory-level scores, discretizes them into predefined sweet-spot zones, and finally applies the resulting structured rewards to downstream optimization tasks.
• Figure 4: Performance on long-term planning tasks. We report Type, GR, SR (%) across two challenging benchmarks.
• Figure 5: Performance on GUI perception tasks. We report accuracy across different interface platforms like Development and Programming, Creative, Office and Operating System.

Theorems & Definitions (4)

• Proposition 3.1: Quality Ordering under Equal Success Rate. Proof in Appendix A
• Proposition 3.2: Projected SNR Improvement. Proof in Appendix A
• proof
• proof