Single-Shot Pruning for Offline Reinforcement Learning

TL;DR

This work tackles the resource cost challenge of deep RL by applying single-shot pruning at initialization to Offline-RL. By using SNIP and GraSP, the authors prune up to $95\%$ of weights before training, integrating seamlessly with BCQ and BC without modifying the RL objective. Experiments on continuous-control tasks demonstrate that pruned networks retain performance in most cases, and memory usage drops by about $4\times$ when stored as sparse tensors. The results highlight a practical pathway to deploy RL on resource-constrained hardware, while raising avenues for robustness to distribution shifts in future work.

Abstract

Deep Reinforcement Learning (RL) is a powerful framework for solving complex real-world problems. Large neural networks employed in the framework are traditionally associated with better generalization capabilities, but their increased size entails the drawbacks of extensive training duration, substantial hardware resources, and longer inference times. One way to tackle this problem is to prune neural networks leaving only the necessary parameters. State-of-the-art concurrent pruning techniques for imposing sparsity perform demonstrably well in applications where data distributions are fixed. However, they have not yet been substantially explored in the context of RL. We close the gap between RL and single-shot pruning techniques and present a general pruning approach to the Offline RL. We leverage a fixed dataset to prune neural networks before the start of RL training. We then run experiments varying the network sparsity level and evaluating the validity of pruning at initialization techniques in continuous control tasks. Our results show that with 95% of the network weights pruned, Offline-RL algorithms can still retain performance in the majority of our experiments. To the best of our knowledge, no prior work utilizing pruning in RL retained performance at such high levels of sparsity. Moreover, pruning at initialization techniques can be easily integrated into any existing Offline-RL algorithms without changing the learning objective.

Figures (4)

• Figure 1: Performance plot of Offline-RL algorithms (BCQ, BC) varying sparsity using SNIP
• Figure 2: Performance plot of Offline-RL algorithms (BCQ, BC) varying sparsity using GraSP
• Figure 3: Visualization of the remaining weights per-layer of the neural networks
• Figure 4: Comparison of memory requirement of sparse and dense models