Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents

Chen Gong, Kecen Li, Jin Yao, Tianhao Wang

TL;DR

This work tackles trajectory-level forgetting in offline reinforcement learning by introducing TrajDeleter, a two-phase method that first makes the forgotten trajectories less valuable and then aligns the unlearned policy with the original on remaining data. To verify forgetting, it proposes TrajAuditor, a lightweight auditing tool built on fine-tuning shadow agents and state perturbations, enabling fast assessment without full retraining. Across six offline RL algorithms and three MuJoCo tasks, TrajDeleter unlearns roughly 93–96% of targeted trajectories with only about 1.5% of the time required for retraining, while maintaining similar performance in deployment; TrajAuditor demonstrates high accuracy (average F1 ≈ 0.88). The methods address privacy, poisoning, and copyright concerns in offline RL and provide a practical pathway for safe data removal with measurable forgetting and efficiency gains.

Abstract

Reinforcement learning (RL) trains an agent from experiences interacting with the environment. In scenarios where online interactions are impractical, offline RL, which trains the agent using pre-collected datasets, has become popular. While this new paradigm presents remarkable effectiveness across various real-world domains, like healthcare and energy management, there is a growing demand to enable agents to rapidly and completely eliminate the influence of specific trajectories from both the training dataset and the trained agents. To meet this problem, this paper advocates Trajdeleter, the first practical approach to trajectory unlearning for offline RL agents. The key idea of Trajdeleter is to guide the agent to demonstrate deteriorating performance when it encounters states associated with unlearning trajectories. Simultaneously, it ensures the agent maintains its original performance level when facing other remaining trajectories. Additionally, we introduce Trajauditor, a simple yet efficient method to evaluate whether Trajdeleter successfully eliminates the specific trajectories of influence from the offline RL agent. Extensive experiments conducted on six offline RL algorithms and three tasks demonstrate that Trajdeleter requires only about 1.5% of the time needed for retraining from scratch. It effectively unlearns an average of 94.8% of the targeted trajectories yet still performs well in actual environment interactions after unlearning. The replication package and agent parameters are available online.

TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents

TL;DR

This work tackles trajectory-level forgetting in offline reinforcement learning by introducing TrajDeleter, a two-phase method that first makes the forgotten trajectories less valuable and then aligns the unlearned policy with the original on remaining data. To verify forgetting, it proposes TrajAuditor, a lightweight auditing tool built on fine-tuning shadow agents and state perturbations, enabling fast assessment without full retraining. Across six offline RL algorithms and three MuJoCo tasks, TrajDeleter unlearns roughly 93–96% of targeted trajectories with only about 1.5% of the time required for retraining, while maintaining similar performance in deployment; TrajAuditor demonstrates high accuracy (average F1 ≈ 0.88). The methods address privacy, poisoning, and copyright concerns in offline RL and provide a practical pathway for safe data removal with measurable forgetting and efficiency gains.

Abstract

Reinforcement learning (RL) trains an agent from experiences interacting with the environment. In scenarios where online interactions are impractical, offline RL, which trains the agent using pre-collected datasets, has become popular. While this new paradigm presents remarkable effectiveness across various real-world domains, like healthcare and energy management, there is a growing demand to enable agents to rapidly and completely eliminate the influence of specific trajectories from both the training dataset and the trained agents. To meet this problem, this paper advocates Trajdeleter, the first practical approach to trajectory unlearning for offline RL agents. The key idea of Trajdeleter is to guide the agent to demonstrate deteriorating performance when it encounters states associated with unlearning trajectories. Simultaneously, it ensures the agent maintains its original performance level when facing other remaining trajectories. Additionally, we introduce Trajauditor, a simple yet efficient method to evaluate whether Trajdeleter successfully eliminates the specific trajectories of influence from the offline RL agent. Extensive experiments conducted on six offline RL algorithms and three tasks demonstrate that Trajdeleter requires only about 1.5% of the time needed for retraining from scratch. It effectively unlearns an average of 94.8% of the targeted trajectories yet still performs well in actual environment interactions after unlearning. The replication package and agent parameters are available online.
Paper Structure (48 sections, 1 theorem, 21 equations, 6 figures, 12 tables, 1 algorithm)

This paper contains 48 sections, 1 theorem, 21 equations, 6 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Offline Reinforcement Learning
  4. Machine Unlearning
  5. Unlearning Scenarios
  6. Principles in Offline RL Unlearning
  7. Problem Setup and Preliminaries
  8. Formal Problem Definition
  9. Challenges and Proposed Solutions
  10. TrajAuditor
  11. Existing Solutions
  12. Technical Details
  13. TrajDeleter
  14. Overview
  15. Technical Details
  16. ...and 33 more sections

Key Result

theorem 1

We assume that the offline dataset includes a diverse range of states. The state distribution generated by any policy is consistently bounded relative to the distribution in the offline dataset. Specifically, denoting the state distribution of the offline dataset as $\mu(s)$, for the state distribut where ${\pi_k}$ denotes a sequence of policies correlated to their respective value functions ${Q^{

Figures (6)

  • Figure 1: An example of offline RL implementations. Initially, the organizer gathers trajectories through interactions with the environments, forming the offline dataset. Then, the agent is trained using this static dataset. Once fully trained, the agent is deployed in real-world applications.
  • Figure 2: The workflow of TrajAuditor. "Shadow agents Training" fine-tunes the original agent and its value function to gather a set of shadow agents. "Value Collections" calculats the value vector of the states of target trajectories from the shadow agents and the target agent. "Auditor," determines whether the trajectories remains in or excluded from the training dataset.
  • Figure 3: The time costs (left) and F1-scores (right) achieved by (1) our proposed TrajAuditor, (2) TrajAuditor without the fine-tuning component, and (3) TrajAuditor excluding state perturbations are presented for comparison.
  • Figure 4: The cumulative returns are averaged over 100 test trajectories, collected using unlearned agents trained with 5 different random seeds. "0.01" and "0.05" represent the unlearning rates. The error bars indicate the standard deviation.
  • Figure 5: The influence of forgetting learning steps, $K$, and the balancing factor, $\lambda$, on the unlearning performance of TrajDeleter. Yellow points denote the average percentages of Positive Predictions (Post. Pred.) by TrajAuditor for unlearned agents, aligning with the left y-axis. Shaded areas denote the standard deviation. Light blue bars illustrate the average returns, corresponding to the right y-axis, with the error bars indicating their standard deviation. The unlearning rate is 0.01.
  • ...and 1 more figures

Theorems & Definitions (1)

  • theorem 1: Interaction convergence TosattoPDR17