Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Actor-Free Continuous Control via Structurally Maximizable Q-Functions

Yigit Korkmaz, Urvi Bhuwania, Ayush Jain, Erdem Bıyık

TL;DR

This work addresses the challenge of performing value-based reinforcement learning in continuous action spaces without an actor. It introduces Q3C, which uses a structurally maximizable Q-function built from control-points, with an action-conditioned Q-value generator and a wire-fitting interpolator to identify the maximizing action directly. Through relevance-based filtering, control-point diversification, and scale-aware normalization, Q3C achieves competitive or superior performance to state-of-the-art actor-critic baselines, especially in environments with restricted action spaces. The results, supported by ablations and visualizations, demonstrate robustness and stability across diverse tasks, and the authors release code to facilitate adoption and further research.

Abstract

Value-based algorithms are a cornerstone of off-policy reinforcement learning due to their simplicity and training stability. However, their use has traditionally been restricted to discrete action spaces, as they rely on estimating Q-values for individual state-action pairs. In continuous action spaces, evaluating the Q-value over the entire action space becomes computationally infeasible. To address this, actor-critic methods are typically employed, where a critic is trained on off-policy data to estimate Q-values, and an actor is trained to maximize the critic's output. Despite their popularity, these methods often suffer from instability during training. In this work, we propose a purely value-based framework for continuous control that revisits structural maximization of Q-functions, introducing a set of key architectural and algorithmic choices to enable efficient and stable learning. We evaluate the proposed actor-free Q-learning approach on a range of standard simulation tasks, demonstrating performance and sample efficiency on par with state-of-the-art baselines, without the cost of learning a separate actor. Particularly, in environments with constrained action spaces, where the value functions are typically non-smooth, our method with structural maximization outperforms traditional actor-critic methods with gradient-based maximization. We have released our code at https://github.com/USC-Lira/Q3C.

Actor-Free Continuous Control via Structurally Maximizable Q-Functions

TL;DR

This work addresses the challenge of performing value-based reinforcement learning in continuous action spaces without an actor. It introduces Q3C, which uses a structurally maximizable Q-function built from control-points, with an action-conditioned Q-value generator and a wire-fitting interpolator to identify the maximizing action directly. Through relevance-based filtering, control-point diversification, and scale-aware normalization, Q3C achieves competitive or superior performance to state-of-the-art actor-critic baselines, especially in environments with restricted action spaces. The results, supported by ablations and visualizations, demonstrate robustness and stability across diverse tasks, and the authors release code to facilitate adoption and further research.

Abstract

Value-based algorithms are a cornerstone of off-policy reinforcement learning due to their simplicity and training stability. However, their use has traditionally been restricted to discrete action spaces, as they rely on estimating Q-values for individual state-action pairs. In continuous action spaces, evaluating the Q-value over the entire action space becomes computationally infeasible. To address this, actor-critic methods are typically employed, where a critic is trained on off-policy data to estimate Q-values, and an actor is trained to maximize the critic's output. Despite their popularity, these methods often suffer from instability during training. In this work, we propose a purely value-based framework for continuous control that revisits structural maximization of Q-functions, introducing a set of key architectural and algorithmic choices to enable efficient and stable learning. We evaluate the proposed actor-free Q-learning approach on a range of standard simulation tasks, demonstrating performance and sample efficiency on par with state-of-the-art baselines, without the cost of learning a separate actor. Particularly, in environments with constrained action spaces, where the value functions are typically non-smooth, our method with structural maximization outperforms traditional actor-critic methods with gradient-based maximization. We have released our code at https://github.com/USC-Lira/Q3C.
Paper Structure (39 sections, 2 theorems, 15 equations, 14 figures, 11 tables, 1 algorithm)

This paper contains 39 sections, 2 theorems, 15 equations, 14 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Q-Learning
  5. Maximization in Q-Learning
  6. Q3C: Q-learning for Continuous Control with Control-points
  7. Function Approximation Using control-points
  8. Reducing Optimization Complexity of control-points
  9. Robustness of Training across Different Tasks
  10. Experiments
  11. Standard Environments
  12. Restricted Environments
  13. Ablations
  14. Visualizing the effect of Q3C components
  15. Conclusion
  16. ...and 24 more sections

Key Result

Proposition 1

Let $\mathcal{A}$ be a compact action set and $s$ be a given state. For any continuous Q-function $Q_s(a) := Q(s, a)$ and any $\epsilon > 0$, there exists a finite set of control-points $\{\hat{a}_1, \dots, \hat{a}_N\} \subset \mathcal{A}$ with corresponding values $y_i = Q_s(a_i)$ such that the wi Hence, the wire-fitting formulation can approximate any continuous $Q(s,\cdot)$ arbitrarily well.

Figures (14)

  • Figure 1: Wire-fitting function. A Q-function represented by an interpolation of learnable control-points is structurally maximized at one of the control-points.
  • Figure 2: Q3C Architecture consists of 3 components: (i) a control-point generator estimates the representative $N$ control-point actions, (ii) a Q-estimator estimates the values of these $N$ points, and (iii) a wire-fitting interpolator estimates the inverse-distance weighted Q-value of the given action.
  • Figure 3: Standard Environments. Q3C is comparable in performance to TD3 and outperforms other actor-free value-based baselines in most environments.
  • Figure 4: Restricted Environments. Q3C consistently outperforms TD3, RBF-DQN, and Wire-Fitting in restricted environments and largely outperforms NAF. Q3C converges at a higher reward with high stability while other algorithms are stuck at local optima.
  • Figure 5: Ablations of Q3C. Every component of Q3C is validated for importance. Individual Q3C components complement each other and combined Q3C model visibly outperforms ablations.
  • ...and 9 more figures

Theorems & Definitions (3)

  • Proposition
  • Proposition
  • proof