Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tractable Offline Learning of Regular Decision Processes

Ahana Deb, Roberto Cipollone, Anders Jonsson, Alessandro Ronca, Mohammad Sadegh Talebi

TL;DR

The paper tackles offline reinforcement learning in Regular Decision Processes, where future dynamics depend on histories via an automaton. It introduces two tractable improvements: a language-based pseudometric framework that organizes pattern testing into a hierarchy of language classes to reduce sample complexity, and Count-Min-Sketch (CMS) to dramatically reduce memory requirements for long planning horizons. The authors provide PAC-style guarantees for both approaches, derive corresponding sample complexity bounds, and validate the methods experimentally against a state-of-the-art automata-learning baseline. The results show that the language-class approach yields smaller models and competitive policies, while CMS offers memory efficiency at the cost of slower runtimes, underscoring practical trade-offs for offline RDP learning. Together, these contributions advance provably correct, scalable offline learning for non-Markovian decision processes with potential impact on complex, history-dependent domains.

Abstract

This work studies offline Reinforcement Learning (RL) in a class of non-Markovian environments called Regular Decision Processes (RDPs). In RDPs, the unknown dependency of future observations and rewards from the past interactions can be captured by some hidden finite-state automaton. For this reason, many RDP algorithms first reconstruct this unknown dependency using automata learning techniques. In this paper, we show that it is possible to overcome two strong limitations of previous offline RL algorithms for RDPs, notably RegORL. This can be accomplished via the introduction of two original techniques: the development of a new pseudometric based on formal languages, which removes a problematic dependency on $L_\infty^\mathsf{p}$-distinguishability parameters, and the adoption of Count-Min-Sketch (CMS), instead of naive counting. The former reduces the number of samples required in environments that are characterized by a low complexity in language-theoretic terms. The latter alleviates the memory requirements for long planning horizons. We derive the PAC sample complexity bounds associated to each of these techniques, and we validate the approach experimentally.

Tractable Offline Learning of Regular Decision Processes

TL;DR

The paper tackles offline reinforcement learning in Regular Decision Processes, where future dynamics depend on histories via an automaton. It introduces two tractable improvements: a language-based pseudometric framework that organizes pattern testing into a hierarchy of language classes to reduce sample complexity, and Count-Min-Sketch (CMS) to dramatically reduce memory requirements for long planning horizons. The authors provide PAC-style guarantees for both approaches, derive corresponding sample complexity bounds, and validate the methods experimentally against a state-of-the-art automata-learning baseline. The results show that the language-class approach yields smaller models and competitive policies, while CMS offers memory efficiency at the cost of slower runtimes, underscoring practical trade-offs for offline RDP learning. Together, these contributions advance provably correct, scalable offline learning for non-Markovian decision processes with potential impact on complex, history-dependent domains.

Abstract

This work studies offline Reinforcement Learning (RL) in a class of non-Markovian environments called Regular Decision Processes (RDPs). In RDPs, the unknown dependency of future observations and rewards from the past interactions can be captured by some hidden finite-state automaton. For this reason, many RDP algorithms first reconstruct this unknown dependency using automata learning techniques. In this paper, we show that it is possible to overcome two strong limitations of previous offline RL algorithms for RDPs, notably RegORL. This can be accomplished via the introduction of two original techniques: the development of a new pseudometric based on formal languages, which removes a problematic dependency on -distinguishability parameters, and the adoption of Count-Min-Sketch (CMS), instead of naive counting. The former reduces the number of samples required in environments that are characterized by a low complexity in language-theoretic terms. The latter alleviates the memory requirements for long planning horizons. We derive the PAC sample complexity bounds associated to each of these techniques, and we validate the approach experimentally.
Paper Structure (31 sections, 7 theorems, 32 equations, 5 figures, 1 table)

This paper contains 31 sections, 7 theorems, 32 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. RL in RDPs
  5. Non-Markovian RL
  6. Offline RL in MDPs.
  7. Preliminaries
  8. Notation
  9. Count-Min-Sketch
  10. Languages and operators
  11. Episodic regular decision processes
  12. Episodic RDPs
  13. Occupancy and distinguishability
  14. Learning RDPs with state-merging algorithms from offline data
  15. Tractable offline learning of RDPs
  16. ...and 16 more sections

Key Result

Theorem 1

AdaCT-H$(\mathcal{D},\delta)$ returns a minimal RDP $\mathbf{R}$ with probability at least $1-4AOQ\delta$ when CMS is used to store the empirical probability distributions of episode suffixes, the statistical test is and the size of the dataset is at least $|\mathcal{D}|\geq\widetilde{\mathcal{O}}(\sqrt{H}/d^\mathsf{b}_{\min} \mu_0)$, where $d^\mathsf{b}_{\min} = \min_{t,q,ao}d^\mathsf{b}_t(q,ao)

Figures (5)

  • Figure 1: T-maze bakker2001rlandlstm with corridor length $N=10$. The observation produced at the initial position $S$ indicates the position of the goal $G$ at the end of the corridor for the current episode.
  • Figure 2: Automaton obtained from the corridor environment. The edges are labelled as [$\textit{action}$, $\textit{observation}$, $\textit{enemy}$].
  • Figure 3: Automaton obtained from T-maze environment with restricted actions. The edges are labelled as [$\textit{action}$, $\textit{observation}$, $\textit{reward}$].
  • Figure 4: Environments.
  • Figure 5: Automaton obtained from T-maze (partially restricted action space), restricted language.

Theorems & Definitions (17)

  • Example 1: T-maze
  • Example 2: RDP for T-maze
  • Definition 1
  • Example 3: Language metric in T-maze
  • Theorem 1
  • Theorem 2
  • Definition 2: Pseudometrics
  • Lemma 3
  • proof
  • Lemma 4
  • ...and 7 more