Fat-to-Thin Policy Optimization: Offline RL with Sparse Policies
Lingwei Zhu, Han Wang, Yukie Nagai
TL;DR
Fat-to-Thin Policy Optimization (FtTPO) addresses offline RL with sparse policies by learning a fat, heavy-tailed proposal policy from logged data and transferring knowledge to a sparse, environment-interacting thin policy. The paper builds a Tsallis-entropy framework that unifies $f$-divergences via the $q$-logarithm and $q^*=2-q$, and introduces practical tools—Munchausen-style recursions and a single-baseline change-of-base approach—to estimate policy ratios across all entropic indices with numerical stability. It develops a theory connecting forward/backward KL, $\chi^n$ divergences, and the $f$-divergence series, enabling regularization without closed-form policies and suggesting applications to safe offline RL and CQL-like regularization. Empirical demonstrations on safety-critical treatment simulations and MuJoCo tasks illustrate the approach, with code available for reproduction, highlighting practical paths to robust sparse-policy optimization in offline settings.
Abstract
Sparse continuous policies are distributions that can choose some actions at random yet keep strictly zero probability for the other actions, which are radically different from the Gaussian. They have important real-world implications, e.g. in modeling safety-critical tasks like medicine. The combination of offline reinforcement learning and sparse policies provides a novel paradigm that enables learning completely from logged datasets a safety-aware sparse policy. However, sparse policies can cause difficulty with the existing offline algorithms which require evaluating actions that fall outside of the current support. In this paper, we propose the first offline policy optimization algorithm that tackles this challenge: Fat-to-Thin Policy Optimization (FtTPO). Specifically, we maintain a fat (heavy-tailed) proposal policy that effectively learns from the dataset and injects knowledge to a thin (sparse) policy, which is responsible for interacting with the environment. We instantiate FtTPO with the general $q$-Gaussian family that encompasses both heavy-tailed and sparse policies and verify that it performs favorably in a safety-critical treatment simulation and the standard MuJoCo suite. Our code is available at \url{https://github.com/lingweizhu/fat2thin}.