Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions
Jiayu Chen, Bhargav Ganguly, Yang Xu, Yongsheng Mei, Tian Lan, Vaneet Aggarwal
TL;DR
This paper provides the first systematic survey of deep generative models for offline policy learning, integrating five mainstream DGMs — Variational Auto-Encoders, Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models — across offline RL and IL. It offers a tutorial-style, unified view by grouping literature by DGM type, clarifying core schemes, and highlighting representative algorithms, extensions, and benchmarks. The work identifies persistent challenges in offline policy learning, such as OOD actions, data coverage, and environment uncertainty, and shows how each DGM addresses these issues through constrained learning, data augmentation, density estimation, trajectory modelling, or diffusion-guided planning. It further discusses open problems, benchmarks, and data/theory directions to guide future research, aiming to catalyze development of more robust DGM-based offline RL/IL methods and potentially generalist agents. Overall, the tutorial-style synthesis serves as a practical reference for researchers to design, compare, and advance DGM-based offline policy learning algorithms with an eye toward data-centric benchmarks, theoretical grounding, and scalable foundation-model approaches.
Abstract
Deep generative models (DGMs) have demonstrated great success across various domains, particularly in generating texts, images, and videos using models trained from offline data. Similarly, data-driven decision-making and robotic control also necessitate learning a generator function from the offline data to serve as the strategy or policy. In this case, applying deep generative models in offline policy learning exhibits great potential, and numerous studies have explored in this direction. However, this field still lacks a comprehensive review and so developments of different branches are relatively independent. In this paper, we provide the first systematic review on the applications of deep generative models for offline policy learning. In particular, we cover five mainstream deep generative models, including Variational Auto-Encoders, Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models, and their applications in both offline reinforcement learning (offline RL) and imitation learning (IL). Offline RL and IL are two main branches of offline policy learning and are widely-adopted techniques for sequential decision-making. Notably, for each type of DGM-based offline policy learning, we distill its fundamental scheme, categorize related works based on the usage of the DGM, and sort out the development process of algorithms in that field. Subsequent to the main content, we provide in-depth discussions on deep generative models and offline policy learning as a summary, based on which we present our perspectives on future research directions. This work offers a hands-on reference for the research progress in deep generative models for offline policy learning, and aims to inspire improved DGM-based offline RL or IL algorithms. For convenience, we maintain a paper list on https://github.com/LucasCJYSDL/DGMs-for-Offline-Policy-Learning.