Neural Policy Style Transfer
Raul Fernandez-Fernandez, Juan G. Victores, Jennifer J. Gago, David Estevez, Carlos Balaguer
TL;DR
NPST tackles transferring behavior style between reinforcement-learning policies while preserving content goals, by learning separate content and style rewards via Maximum Entropy IRL and combining them through a generated policy $\mathcal{G}$ initialized from a style policy. The framework uses three DQNs to encode content, style, and generated actions, with content loss $\mathcal{L}_{content}$ and style loss $\mathcal{L}_{style}$ guiding backpropagation and L-BFGS-B optimization for style alignment. Experiments in Catch-ball and Grid-world paint demonstrate that NPST can imprint distinct styles onto policies across multiple architectures, producing interpretable shifts in action selection and state visitation while preserving the underlying task objective. Although style transfer can reduce raw win rates, it yields policies that enact the intended style, with potential applications in socially aware robots and adaptable control systems. The work presents NPST as a computationally efficient approach, avoiding extra loss networks and enabling direct policy-level style adaptation in real-world robotics contexts.
Abstract
Style Transfer has been proposed in a number of fields: fine arts, natural language processing, and fixed trajectories. We scale this concept up to control policies within a Deep Reinforcement Learning infrastructure. Each network is trained to maximize the expected reward, which typically encodes the goal of an action, and can be described as the content. The expressive power of deep neural networks enables encoding a secondary task, which can be described as the style. The Neural Policy Style Transfer (NPST) algorithm is proposed to transfer the style of one policy to another, while maintaining the content of the latter. Different policies are defined via Deep Q-Network architectures. These models are trained using demonstrations through Inverse Reinforcement Learning. Two different sets of user demonstrations are performed, one for content and other for style. Different styles are encoded as defined by user demonstrations. The generated policy is the result of feeding a content policy and a style policy to the NPST algorithm. Experiments are performed in a catch-ball game inspired by the Deep Reinforcement Learning classical Atari games; and a real-world painting scenario with a full-sized humanoid robot, based on previous works of the authors. The implementation of three different Q-Network architectures (Shallow, Deep and Deep Recurrent Q-Network) to encode the policies within the NPST framework is proposed and the results obtained in the experiments with each of these architectures compared.