OffSim: Offline Simulator for Model-based Offline Inverse Reinforcement Learning

TL;DR

OffSim introduces a model-based offline IRL framework that learns an offline simulator combining a high-entropy transition model with an IRL-derived reward from expert trajectories, enabling effective policy learning entirely offline. The method is extended with OffSim^+ to incorporate a margin constraint for multi-dataset exploration, improving robustness and exploration when combining expert and diverse data. Through two-stage training and SAC-based policy optimization, OffSim achieves strong performance on MuJoCo tasks and demonstrates compatibility with multiple RL backbones. The results indicate that jointly modeling dynamics and reward in an offline simulator yields superior generalization and sample efficiency for offline IRL, with practical benefits for safety-critical domains.

Abstract

Reinforcement learning algorithms typically utilize an interactive simulator (i.e., environment) with a predefined reward function for policy training. Developing such simulators and manually defining reward functions, however, is often time-consuming and labor-intensive. To address this, we propose an Offline Simulator (OffSim), a novel model-based offline inverse reinforcement learning (IRL) framework, to emulate environmental dynamics and reward structure directly from expert-generated state-action trajectories. OffSim jointly optimizes a high-entropy transition model and an IRL-based reward function to enhance exploration and improve the generalizability of the learned reward. Leveraging these learned components, OffSim can subsequently train a policy offline without further interaction with the real environment. Additionally, we introduce OffSim$^+$, an extension that incorporates a marginal reward for multi-dataset settings to enhance exploration. Extensive MuJoCo experiments demonstrate that OffSim achieves substantial performance gains over existing offline IRL methods, confirming its efficacy and robustness.

Figures (1)

• Figure 1: Conceptual illustration of OffSim. While online reinforcement learning interacts with the environment and updates the policy through direct exploration, our proposed framework emulates an environment from expert data.