Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Relative Entropy to Minimax: A Unified Framework for Coverage in MDPs

Xihe Gu, Urbashi Mitra, Tara Javidi

TL;DR

A gradient-based algorithm is developed that actively steers the induced occupancy toward a desired coverage pattern, and it is shown that as $\rho$ increases, the resulting exploration strategy increasingly emphasizes the least-explored state--action pairs, recovering worst-case coverage behavior in the limit.

Abstract

Targeted and deliberate exploration of state--action pairs is essential in reward-free Markov Decision Problems (MDPs). More precisely, different state-action pairs exhibit different degree of importance or difficulty which must be actively and explicitly built into a controlled exploration strategy. To this end, we propose a weighted and parameterized family of concave coverage objectives, denoted by $U_ρ$, defined directly over state--action occupancy measures. This family unifies several widely studied objectives within a single framework, including divergence-based marginal matching, weighted average coverage, and worst-case (minimax) coverage. While the concavity of $U_ρ$ captures the diminishing return associated with over-exploration, the simple closed form of the gradient of $U_ρ$ enables an explicit control to prioritize under-explored state--action pairs. Leveraging this structure, we develop a gradient-based algorithm that actively steers the induced occupancy toward a desired coverage pattern. Moreover, we show that as $ρ$ increases, the resulting exploration strategy increasingly emphasizes the least-explored state--action pairs, recovering worst-case coverage behavior in the limit.

From Relative Entropy to Minimax: A Unified Framework for Coverage in MDPs

TL;DR

A gradient-based algorithm is developed that actively steers the induced occupancy toward a desired coverage pattern, and it is shown that as increases, the resulting exploration strategy increasingly emphasizes the least-explored state--action pairs, recovering worst-case coverage behavior in the limit.

Abstract

Targeted and deliberate exploration of state--action pairs is essential in reward-free Markov Decision Problems (MDPs). More precisely, different state-action pairs exhibit different degree of importance or difficulty which must be actively and explicitly built into a controlled exploration strategy. To this end, we propose a weighted and parameterized family of concave coverage objectives, denoted by , defined directly over state--action occupancy measures. This family unifies several widely studied objectives within a single framework, including divergence-based marginal matching, weighted average coverage, and worst-case (minimax) coverage. While the concavity of captures the diminishing return associated with over-exploration, the simple closed form of the gradient of enables an explicit control to prioritize under-explored state--action pairs. Leveraging this structure, we develop a gradient-based algorithm that actively steers the induced occupancy toward a desired coverage pattern. Moreover, we show that as increases, the resulting exploration strategy increasingly emphasizes the least-explored state--action pairs, recovering worst-case coverage behavior in the limit.
Paper Structure (21 sections, 7 theorems, 79 equations, 1 algorithm)

This paper contains 21 sections, 7 theorems, 79 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Controlled Markov Chain Model
  4. Policies
  5. Induced Occupancy Measures
  6. Exploration Objective
  7. A General Class of Coverage Objectives
  8. Analysis: From Relative Entropy to Minimax Exploration
  9. Special case: $\rho = 1$ (KL divergence)
  10. Special case: $\rho = 2$ (Average relative coverage)
  11. Special case: $\rho \to \infty$ (Worst-case coverage)
  12. Optimization Algorithm
  13. State Distribution Estimate
  14. Smoothed Optimization
  15. Algorithm Design
  16. ...and 6 more sections

Key Result

Lemma 1

Define the normalized weights When $\rho = 1$, maximizing the coverage objective over $d$ is equivalent to minimizing the KL divergence $\mathrm{KL}(\bar{\mu} \,\|\, d)$.

Theorems & Definitions (12)

  • Lemma 1
  • proof
  • Lemma 2: Average relative coverage
  • Lemma 3
  • proof
  • Lemma 4
  • Lemma 5
  • Theorem 1
  • proof
  • Lemma 6: Bound of empirical occupancy
  • ...and 2 more