Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functions

TL;DR

This work tackles suboptimality in deterministic policy gradients caused by non-convex Q-landscapes. It introduces SAVO, a plug-in actor-architecture that trains multiple gradient-based actors and uses successive surrogate Q-functions to prune low-value regions, thereby guiding gradient ascent toward higher-value actions. Across restricted locomotion, dexterous manipulation, and large discrete-action recommender tasks, SAVO reduces local optima and outperforms single-actor baselines and related multi-actor or sampling-based methods. The approach preserves TD3/SAC sample efficiency while improving final performance, suggesting a practical, scalable path to robust policy optimization in complex control and decision-making settings.

Abstract

In reinforcement learning, off-policy actor-critic methods like DDPG and TD3 use deterministic policy gradients: the Q-function is learned from environment data, while the actor maximizes it via gradient ascent. We observe that in complex tasks such as dexterous manipulation and restricted locomotion with mobility constraints, the Q-function exhibits many local optima, making gradient ascent prone to getting stuck. To address this, we introduce SAVO, an actor architecture that (i) generates multiple action proposals and selects the one with the highest Q-value, and (ii) approximates the Q-function repeatedly by truncating poor local optima to guide gradient ascent more effectively. We evaluate tasks such as restricted locomotion, dexterous manipulation, and large discrete-action space recommender systems and show that our actor finds optimal actions more frequently and outperforms alternate actor architectures.

Figures (32)

• Figure 1: In continuous control tasks, we visualize trained TD3 Q-values at a fixed state $s_f$ over the full action space (projected to 2D), $Q(s, a | s=s_f)$. In Inverted-Double-Pendulum (left) and Hopper (middle) with action ranges restricted due to immobility, Q-landscapes have multiple local optima. In a large discrete-action recommendation task (right), local peaks correspond to real items (black dots). In such non-convex Q-landscapes, gradient-based actors often converge to suboptimal actions.
• Figure 2: An actor $\mu$ trained with gradient ascent on a challenging Q-landscape gets stuck in local optima. Our approach learns a sequence of surrogates $\Psi_i$ of the Q-function that successively prune out the Q-landscape below the current best Q-values, resulting in fewer local optima. Thus, the actors $\nu_i$ trained to ascend on these surrogates produce actions with a more optimal Q-value.
• Figure 3: Non-convex Q-landscape in Inverted-Pendulum-Restricted leads to the TD3 actor converging at a local optimum $a_0$ with large suboptimality.
• Figure 4: SAVO Architecture. (left) Q-network is unchanged. (center) Instead of a single actor, we learn a sequence of actors and surrogate networks connected via action predictions. (right) Conditioning on previous actions is done with the help of a deep-set summarizer and FiLM modulation.
• Figure 5: While $\Psi$ (left) has flat surfaces, $\hat{\Psi}$ (right) smoothens the function to allow non-zero gradients to flow into the actor towards better optima in Inverted-Pendulum-Restricted.

Theorems & Definitions (6)

• Theorem 4.1: Convergence of Policy Iteration with Maximizer Actor
• Theorem 4.2
• Theorem 9.1: Convergence of Policy Iteration with Maximizer Actor
• proof
• Theorem 10.1
• proof