Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning

Gaurav Chaudhary, Laxmidhar Behera

TL;DR

Offline reinforcement learning suffers from reward annotation scarcity. ReLOAD distills intrinsic rewards from expert transitions using Random Network Distillation, assigning rewards via embedding discrepancies between a fixed target and a predictor trained on expert data. A formal theorem shows that the predictor-target error distinguishes expert-like transitions, enabling imitation-guided policy learning from reward-free datasets. Empirical results on D4RL locomotion, AntMaze, and Adroit demonstrate competitive performance with reward-based offline RL baselines while remaining computationally efficient and robust to limited expert data.

Abstract

Offline Reinforcement Learning (RL) aims to learn effective policies from a static dataset without requiring further agent-environment interactions. However, its practical adoption is often hindered by the need for explicit reward annotations, which can be costly to engineer or difficult to obtain retrospectively. To address this, we propose ReLOAD (Reinforcement Learning with Offline Reward Annotation via Distillation), a novel reward annotation framework for offline RL. Unlike existing methods that depend on complex alignment procedures, our approach adapts Random Network Distillation (RND) to generate intrinsic rewards from expert demonstrations using a simple yet effective embedding discrepancy measure. First, we train a predictor network to mimic a fixed target network's embeddings based on expert state transitions. Later, the prediction error between these networks serves as a reward signal for each transition in the static dataset. This mechanism provides a structured reward signal without requiring handcrafted reward annotations. We provide a formal theoretical construct that offers insights into how RND prediction errors effectively serve as intrinsic rewards by distinguishing expert-like transitions. Experiments on the D4RL benchmark demonstrate that ReLOAD enables robust offline policy learning and achieves performance competitive with traditional reward-annotated methods.

From Novelty to Imitation: Self-Distilled Rewards for Offline Reinforcement Learning

TL;DR

Offline reinforcement learning suffers from reward annotation scarcity. ReLOAD distills intrinsic rewards from expert transitions using Random Network Distillation, assigning rewards via embedding discrepancies between a fixed target and a predictor trained on expert data. A formal theorem shows that the predictor-target error distinguishes expert-like transitions, enabling imitation-guided policy learning from reward-free datasets. Empirical results on D4RL locomotion, AntMaze, and Adroit demonstrate competitive performance with reward-based offline RL baselines while remaining computationally efficient and robust to limited expert data.

Abstract

Offline Reinforcement Learning (RL) aims to learn effective policies from a static dataset without requiring further agent-environment interactions. However, its practical adoption is often hindered by the need for explicit reward annotations, which can be costly to engineer or difficult to obtain retrospectively. To address this, we propose ReLOAD (Reinforcement Learning with Offline Reward Annotation via Distillation), a novel reward annotation framework for offline RL. Unlike existing methods that depend on complex alignment procedures, our approach adapts Random Network Distillation (RND) to generate intrinsic rewards from expert demonstrations using a simple yet effective embedding discrepancy measure. First, we train a predictor network to mimic a fixed target network's embeddings based on expert state transitions. Later, the prediction error between these networks serves as a reward signal for each transition in the static dataset. This mechanism provides a structured reward signal without requiring handcrafted reward annotations. We provide a formal theoretical construct that offers insights into how RND prediction errors effectively serve as intrinsic rewards by distinguishing expert-like transitions. Experiments on the D4RL benchmark demonstrate that ReLOAD enables robust offline policy learning and achieves performance competitive with traditional reward-annotated methods.

Paper Structure

This paper contains 31 sections, 1 theorem, 19 equations, 2 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Offline RL with Reward Annotations
  4. Imitation Learning (IL)
  5. Reward Inference in Offline RL
  6. Preliminaries
  7. Offline RL
  8. Offline RL
  9. Random Network Distillation (RND)
  10. Formal Justification of the Novelty Reward
  11. Method
  12. Expert Embedding Alignment:
  13. Intrinsic Reward Assignment via Embedding Discrepancy:
  14. Offline Policy Learning with Distilled Rewards:
  15. Reward Squashing for Stability:
  16. ...and 16 more sections

Key Result

Theorem 1

Let $D_E$ and $D_U$ be probability distributions over state transitions $X$ (where $x = (s, s')$ represents a transition), and the support of $D_E$ is a proper subset of $D_U$, i.e., $\operatorname{supp}(D_E) \subsetneq \operatorname{supp}(D_U)$. Define $f: X \to \mathbb{R}^K$ as a fixed, randomly i Let the expected prediction error on $D_E$ be: Assume that: Then, the expected prediction error o

Figures (2)

  • Figure 1: Overview of our method, ReLOAD. Transitions from expert demonstrations are used to train a predictor network to align with a fixed target network. When the unlabeled transitions are processed through target and predictor networks, the embedding discrepancy on an unlabeled dataset serves as an intrinsic reward to guide offline policy learning.
  • Figure 2: Comparison of IQL and ReLOAD on D4RL Locomotion Tasks across varying fractions of offline data (10%, 25%, 50%, 100%). ReLOAD consistently outperforms IQL, particularly in Hopper and Walker2d environments. Bars indicate mean normalized scores with standard deviation over 10 random seeds.

Theorems & Definitions (1)

  • Theorem 1