Application of linear regression and quasi-Newton methods to the deep reinforcement learning in continuous action cases

TL;DR

The paper addresses the challenge of applying linear-regression (LR) updates to deep reinforcement learning in continuous action spaces. It introduces DLS-DDPG, which blends a Least Squares FQI–style LR update for the critic with a quasi-Newton optimization to compute an optimal action $o$ for the actor LR update, using $o$ as both the action and training data. Numerical results on MuJoCo tasks show that DLS-DDPG can improve learning in several environments, with significant gains largely driven by the optimal-action choosing (OAC) component; regularization is necessary to prevent weight divergence. This work demonstrates a viable path to integrating LR methods with off-policy continuous-control DRL and highlights practical considerations for stability and hyperparameter tuning, pointing to future work with TD3/SAC-style extensions.

Abstract

The linear regression (LR) method offers the advantage that optimal parameters can be calculated relatively easily, although its representation capability is limited than that of the deep learning technique. To improve deep reinforcement learning, the Least Squares Deep Q Network (LS-DQN) method was proposed by Levine et al., which combines Deep Q Network (DQN) with LR method. However, the LS-DQN method assumes that the actions are discrete. In this study, we propose the Double Least Squares Deep Deterministic Policy Gradient (DLS-DDPG) method to address this limitation. This method combines the LR method with the Deep Deterministic Policy Gradient (DDPG) technique, one of the representative deep reinforcement learning algorithms for continuous action cases. For the LR update of the critic network, DLS-DDPG uses an algorithm similar to the Fitted Q iteration, the method which LS-DQN adopted. In addition, we calculated the optimal action using the quasi-Newton method and used it as both the agent's action and the training data for the LR update of the actor network. Numerical experiments conducted in MuJoCo environments showed that the proposed method improved performance at least in some tasks, although there are difficulties such as the inability to make the regularization terms small.

Figures (9)

• Figure 1: Learning curves of the cases that did not use DDPG update for the critic, for six MuJoCo tasks. The case where $\beta_c$ and $\beta ' _c$ varied according to eqs. (\ref{['DLSDDPG_beta_c']}) and (\ref{['DLSDDPG_beta_prime_c']}) and Table \ref{['hyperparameters']} (red), and the case where $\beta_c = \beta ' _c = 10^{-3}$ (purple) were investigated.
• Figure 2: Learning curves of DLS-DDPG with OAC(red) and AAC(purple), and the cases that did not use DDPG update for the actor with OAC(green) and AAC(orange), for six MuJoCo tasks
• Figure 3: Learning curves of DLS-DDPG(red), DDPG with LR update of actor(green) or critic(purple), our DDPG(blue) and the usual DDPG(black), for six MuJoCo tasks
• Figure 4: Learning curves of DLS-DDPG(red), DDPG with LR update of the critic and OAC (cyan), and that with LR update of the critic and AAC (purple), for four MuJoCo tasks
• Figure 5: Learning curves of DDPG with LR update of the actor and OAC (green), DDPG with OAC (orange), and that with AAC (blue), for four MuJoCo tasks