Can We Optimize Deep RL Policy Weights as Trajectory Modeling?
Hongyao Tang
TL;DR
The paper investigates whether the trajectory of policy network weights during DRL training can be modeled and leveraged as a data modality. It introduces TIPL, a Transformer-based implicit policy learner that autoregressively predicts future policy weights from historical weights, using Temporal SVD to obtain compact tokens and training with an MSE objective. Trained on PPO trajectories from two MuJoCo tasks, TIPL demonstrates the ability to fit the implicit dynamics of policy learning and to generate plausible next weights, with corresponding return estimates aligning with true outcomes. This approach suggests a data-driven alternative to conventional gradient-based optimization and highlights both the potential and limitations of weight-space trajectory modeling for policy optimization.
Abstract
Learning the optimal policy from a random network initialization is the theme of deep Reinforcement Learning (RL). As the scale of DRL training increases, treating DRL policy network weights as a new data modality and exploring the potential becomes appealing and possible. In this work, we focus on the policy learning path in deep RL, represented by the trajectory of network weights of historical policies, which reflects the evolvement of the policy learning process. Taking the idea of trajectory modeling with Transformer, we propose Transformer as Implicit Policy Learner (TIPL), which processes policy network weights in an autoregressive manner. We collect the policy learning path data by running independent RL training trials, with which we then train our TIPL model. In the experiments, we demonstrate that TIPL is able to fit the implicit dynamics of policy learning and perform the optimization of policy network by inference.