Linear Function Approximation as a Computationally Efficient Method to Solve Classical Reinforcement Learning Challenges

TL;DR

The paper tackles whether Linear Function Approximation combined with Natural Policy Gradient ($LFA$-$NPG$) can outperform neural-network based policy optimizers like TRPO and PPO on low-dimensional reinforcement learning tasks. It implements a log-linear policy with a linear critic and uses the Fisher Information-based natural gradient to update policy parameters, evaluating on CartPole and Acrobot with handcrafted feature mappings $\phi$ to form the state representation. Empirically, $LFA$-$NPG$ matches or surpasses TRPO/PPO in final rewards and achieves substantially faster convergence, particularly in sparse-reward settings, while also exhibiting robustness to noise. The work argues for adopting $LFA$-$NPG$ as a practical alternative in simple state-action spaces and outlines future directions toward continuous actions and automated feature selection.

Abstract

Neural Network based approximations of the Value function make up the core of leading Policy Based methods such as Trust Regional Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). While this adds significant value when dealing with very complex environments, we note that in sufficiently low State and action space environments, a computationally expensive Neural Network architecture offers marginal improvement over simpler Value approximation methods. We present an implementation of Natural Actor Critic algorithms with actor updates through Natural Policy Gradient methods. This paper proposes that Natural Policy Gradient (NPG) methods with Linear Function Approximation as a paradigm for value approximation may surpass the performance and speed of Neural Network based models such as TRPO and PPO within these environments. Over Reinforcement Learning benchmarks Cart Pole and Acrobot, we observe that our algorithm trains much faster than complex neural network architectures, and obtains an equivalent or greater result. This allows us to recommend the use of NPG methods with Linear Function Approximation over TRPO and PPO for both traditional and sparse reward low dimensional problems.

Figures (5)

• Figure 1: Simplified RL procedure
• Figure 2: Visualization of CartPole and Acrobot
• Figure 3: Optimizing the set of function approximators $\phi$ for LFA-NPG
• Figure 4: Comparing LFA-NPG with Flagship Neural Networks
• Figure 5: LFA-NPG, TRPO, and PPO Robustness Analysis