Residual Q-Learning: Offline and Online Policy Customization without Value

TL;DR

The paper addresses how to customize an imitative policy to downstream tasks without uncovering the prior reward. It frames policy customization as an MDP with a reward that linearly combines the unknown inherent reward and a known add-on, and introduces Residual Q-Learning (RQL) that learns a residual Q-function on top of the prior policy's value, enabling principled offline and online customization. It delivers three practical algorithms—Residual Soft Q-Learning, Residual Soft Actor-Critic, and Residual Maximum-Entropy MCTS—and demonstrates that customized policies achieve higher add-on performance while preserving basic-task behavior across diverse domains and settings. The work provides a principled pathway for integrating imitation with task-specific objectives, with practical impact for robotics and autonomous systems where hand-crafted rewards are difficult and safety or behavior preferences are critical.

Abstract

Imitation Learning (IL) is a widely used framework for learning imitative behavior from demonstrations. It is especially appealing for solving complex real-world tasks where handcrafting reward function is difficult, or when the goal is to mimic human expert behavior. However, the learned imitative policy can only follow the behavior in the demonstration. When applying the imitative policy, we may need to customize the policy behavior to meet different requirements coming from diverse downstream tasks. Meanwhile, we still want the customized policy to maintain its imitative nature. To this end, we formulate a new problem setting called policy customization. It defines the learning task as training a policy that inherits the characteristics of the prior policy while satisfying some additional requirements imposed by a target downstream task. We propose a novel and principled approach to interpret and determine the trade-off between the two task objectives. Specifically, we formulate the customization problem as a Markov Decision Process (MDP) with a reward function that combines 1) the inherent reward of the demonstration; and 2) the add-on reward specified by the downstream task. We propose a novel framework, Residual Q-learning, which can solve the formulated MDP by leveraging the prior policy without knowing the inherent reward or value function of the prior policy. We derive a family of residual Q-learning algorithms that can realize offline and online policy customization, and show that the proposed algorithms can effectively accomplish policy customization tasks in various environments. Demo videos and code are available on our website: https://sites.google.com/view/residualq-learning.

Figures (4)

• Figure 1: Learning curves in the Parking environment for different algorithms, including residual soft actor-critic, soft actor-critic with divergence-augmented reward (Appendix \ref{['sec:appendix-ablation-KL']}), and soft actor-critic with likelihood-augmented reward (Appendix \ref{['sec:appendix-ablation-ll']}). We plot the curves of the non-violation rate and success rate over the number of episodic steps in subplots (a) and (b) respectively. The error bands indicate standard deviations computed over four trials with different random seeds.
• Figure 2: Representative examples from the Parking environment comparing the results of executing the IL prior and the RQL policy customized from the IL prior.
• Figure 3: Representative examples from the Parking environment comparing the results of executing RL prior, RL full policy, and policies customized from RL prior with different approaches.
• Figure 4: (a) The trajectory of the Ant robot on the $x$ and $y$ axis. (b) The trajectory of the Humanoid robot on the $x$ and $y$ axis. (c) The trajectory of the top of the Hopper robot on the $x$ and $z$ axis.