Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reinforcement Learning with $ω$-Regular Objectives and Constraints

Dominik Wagner, Leon Witzman, Luke Ong

TL;DR

This work develops a model-based RL algorithm based on linear programming, which in the limit produces a policy maximising the probability of satisfying an $\omega$-regular objective while also adhering to $\omega$-regular constraints within specified thresholds.

Abstract

Reinforcement learning (RL) commonly relies on scalar rewards with limited ability to express temporal, conditional, or safety-critical goals, and can lead to reward hacking. Temporal logic expressible via the more general class of $ω$-regular objectives addresses this by precisely specifying rich behavioural properties. Even still, measuring performance by a single scalar (be it reward or satisfaction probability) masks safety-performance trade-offs that arise in settings with a tolerable level of risk. We address both limitations simultaneously by combining $ω$-regular objectives with explicit constraints, allowing safety requirements and optimisation targets to be treated separately. We develop a model-based RL algorithm based on linear programming, which in the limit produces a policy maximising the probability of satisfying an $ω$-regular objective while also adhering to $ω$-regular constraints within specified thresholds. Furthermore, we establish a translation to constrained limit-average problems with optimality-preserving guarantees.

Reinforcement Learning with $ω$-Regular Objectives and Constraints

TL;DR

This work develops a model-based RL algorithm based on linear programming, which in the limit produces a policy maximising the probability of satisfying an -regular objective while also adhering to -regular constraints within specified thresholds.

Abstract

Reinforcement learning (RL) commonly relies on scalar rewards with limited ability to express temporal, conditional, or safety-critical goals, and can lead to reward hacking. Temporal logic expressible via the more general class of -regular objectives addresses this by precisely specifying rich behavioural properties. Even still, measuring performance by a single scalar (be it reward or satisfaction probability) masks safety-performance trade-offs that arise in settings with a tolerable level of risk. We address both limitations simultaneously by combining -regular objectives with explicit constraints, allowing safety requirements and optimisation targets to be treated separately. We develop a model-based RL algorithm based on linear programming, which in the limit produces a policy maximising the probability of satisfying an -regular objective while also adhering to -regular constraints within specified thresholds. Furthermore, we establish a translation to constrained limit-average problems with optimality-preserving guarantees.

Paper Structure

This paper contains 21 sections, 3 theorems, 9 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work.
  4. Problem Statement
  5. $\omega$-regular specifications.
  6. Policies.
  7. Constrained Optimisation Problem.
  8. Expressivity.
  9. Planning and Optimal Policy
  10. Correctness and Type of Optimal Policies.
  11. Model-Based Reinforcement Learning Algorithm
  12. Translation To Constrained Limit Average Problem
  13. Failure of Independent Translation.
  14. Construction.
  15. Reward Assignment.
  16. ...and 6 more sections

Key Result

Theorem 1

Optimal policies which are a convex combination of two stochastic, stationary policies always exist.

Figures (2)

  • Figure 1: Example MDP where the objective is to reach target states (labelled $t$) whilst avoiding unsafe states (labelled $u$). The LTL objective $(\square \neg u) \land (\diamond t)$ defines runs where unsafe states are always avoided and a target state is eventually reached. Its optimal policy selects action $b$, resulting in the unsafe state with $40\%$ probability. Selecting $a$ is always safe and reaches the target state with $50\%$ probability.
  • Figure 2: Deterministic MDP of \ref{['ex:memory']}.

Theorems & Definitions (7)

  • Example 1
  • Example 2
  • Theorem 1
  • Theorem 2
  • Theorem 3: Convergence
  • Definition 1
  • Example 3