Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Action-Free Offline-to-Online RL via Discretised State Policies

Natinael Solomon Neggatu, Jeremie Houssineau, Giovanni Montana

TL;DR

This paper tackles action-free offline-to-online RL, where offline data lacks actions and contains only $ (s,r,s') $ tuples. It introduces OSO-DecQN, a discretised state-difference value learner that pre-trains state policies without action labels, and a guided online-learning pipeline that translates predicted state changes into actions via a lightweight IDM. The approach combines discretisation with conservative regularisation to prevent overestimation and ensure state reachability, and couples this with a policy-switching mechanism to guide online exploration. Empirical results across diverse benchmarks show faster convergence and improved asymptotic performance, with the discretisation and regularisation components identified as critical for effectiveness and scalability to high-dimensional state spaces.

Abstract

Most existing offline RL methods presume the availability of action labels within the dataset, but in many practical scenarios, actions may be missing due to privacy, storage, or sensor limitations. We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of $(s,r,s')$ tuples and later leverage this knowledge during online interaction. To address this challenge, we propose learning state policies that recommend desirable next-state transitions rather than actions. Our contributions are twofold. First, we introduce a simple yet novel state discretisation transformation and propose Offline State-Only DecQN (\algo), a value-based algorithm designed to pre-train state policies from action-free data. \algo{} integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, we propose a novel mechanism for guided online learning that leverages these pre-trained state policies to accelerate the learning of online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that our approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical to its effectiveness.

Action-Free Offline-to-Online RL via Discretised State Policies

TL;DR

This paper tackles action-free offline-to-online RL, where offline data lacks actions and contains only tuples. It introduces OSO-DecQN, a discretised state-difference value learner that pre-trains state policies without action labels, and a guided online-learning pipeline that translates predicted state changes into actions via a lightweight IDM. The approach combines discretisation with conservative regularisation to prevent overestimation and ensure state reachability, and couples this with a policy-switching mechanism to guide online exploration. Empirical results across diverse benchmarks show faster convergence and improved asymptotic performance, with the discretisation and regularisation components identified as critical for effectiveness and scalability to high-dimensional state spaces.

Abstract

Most existing offline RL methods presume the availability of action labels within the dataset, but in many practical scenarios, actions may be missing due to privacy, storage, or sensor limitations. We formalise the setting of action-free offline-to-online RL, where agents must learn from datasets consisting solely of tuples and later leverage this knowledge during online interaction. To address this challenge, we propose learning state policies that recommend desirable next-state transitions rather than actions. Our contributions are twofold. First, we introduce a simple yet novel state discretisation transformation and propose Offline State-Only DecQN (\algo), a value-based algorithm designed to pre-train state policies from action-free data. \algo{} integrates the transformation to scale efficiently to high-dimensional problems while avoiding instability and overfitting associated with continuous state prediction. Second, we propose a novel mechanism for guided online learning that leverages these pre-trained state policies to accelerate the learning of online agents. Together, these components establish a scalable and practical framework for leveraging action-free datasets to accelerate online RL. Empirical results across diverse benchmarks demonstrate that our approach improves convergence speed and asymptotic performance, while analyses reveal that discretisation and regularisation are critical to its effectiveness.
Paper Structure (50 sections, 5 theorems, 29 equations, 7 figures, 14 tables, 1 algorithm)

This paper contains 50 sections, 5 theorems, 29 equations, 7 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Offline RL
  4. State-only RL
  5. Single agent action value decomposition
  6. Background
  7. Preliminaries
  8. Decoupled Q-Networks
  9. Methodology
  10. Discretising the target state space
  11. OSO-DecQN
  12. Regularisation
  13. Guided online learning
  14. Leveraging Q-values
  15. Training an IDM
  16. ...and 35 more sections

Key Result

Theorem 1

If the discretised increment space $\Delta s$ is partitioned into $k$ evenly spaced bins per coordinate, then where $V^*$ and $V_D^*$ are the optimal value functions of the original and discretised MDPs. Here $M$ denotes the state dimension and $H \coloneqq \delta_s^{\mathrm{max}}-\delta_s^{\mathrm{min}}$ denotes the range of the per-coordinate mean increments. The required assumptions, and the p

Figures (7)

  • Figure 1: Online learning curves comparing the performance of guided online learning with state policies pre-trained on datasets of different qualities against baselines over 1M timesteps. The solid line corresponds to the mean normalised return across 5 seeds with the shaded area corresponding to 1 standard deviation away from the mean.
  • Figure 2: Comparison of Af-guide zhu2023guiding against TD3 baseline. The solid line corresponds to the mean normalised return across 5 seeds with the shaded area corresponding to 1 s.d.
  • Figure 3: Comparison of fixed versus linearly annealed $\beta$ in guided online learning for Walker2D and Hopper.
  • Figure 4: A look at the IDM loss function during online training
  • Figure 5: Online learning curves on D4RL, each with corresponding legend. We compare our method of guided learning to the performance of an online agent trained using SAC, without guidance over 1M timesteps. The solid line corresponds to the mean normalised return across 5 seeds with the shaded area corresponding to 1 standard deviation away from the mean.
  • ...and 2 more figures

Theorems & Definitions (8)

  • Theorem 1: Value bound under $k$ evenly spaced bins in $\Delta s$
  • Lemma 1: Control of expectation differences via TV and KL
  • Lemma 2: KL bound for discretised actions
  • proof
  • Lemma 3: $\Delta s$--binning bound in continuous space
  • proof
  • Theorem 2: Value bound under $k$ evenly spaced bins in $\Delta s$
  • proof