Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversity

Felix Schur

Abstract

Can latent actions and environment dynamics be recovered from offline trajectories when actions are never observed? We study this question in a setting where trajectories are action-free but tagged with demonstrator identity. We assume that each demonstrator follows a distinct policy, while the environment dynamics are shared across demonstrators and identity affects the next observation only through the chosen action. Under these assumptions, the conditional next-observation distribution $p(o_{t+1}\mid o_t,e)$ is a mixture of latent action-conditioned transition kernels with demonstrator-specific mixing weights. We show that this induces, for each state, a column-stochastic nonnegative matrix factorization of the observable conditional distribution. Using sufficiently scattered policy diversity and rank conditions, we prove that the latent transitions and demonstrator policies are identifiable up to permutation of the latent action labels. We extend the result to continuous observation spaces via a Gram-determinant minimum-volume criterion, and show that continuity of the transition map over a connected state space upgrades local permutation ambiguities to a single global permutation. A small amount of labeled action data then suffices to fix this final ambiguity. These results establish demonstrator diversity as a principled source of identifiability for learning latent actions and dynamics from offline RL data.

Identifying Latent Actions and Dynamics from Offline Data via Demonstrator Diversity

Abstract

Can latent actions and environment dynamics be recovered from offline trajectories when actions are never observed? We study this question in a setting where trajectories are action-free but tagged with demonstrator identity. We assume that each demonstrator follows a distinct policy, while the environment dynamics are shared across demonstrators and identity affects the next observation only through the chosen action. Under these assumptions, the conditional next-observation distribution is a mixture of latent action-conditioned transition kernels with demonstrator-specific mixing weights. We show that this induces, for each state, a column-stochastic nonnegative matrix factorization of the observable conditional distribution. Using sufficiently scattered policy diversity and rank conditions, we prove that the latent transitions and demonstrator policies are identifiable up to permutation of the latent action labels. We extend the result to continuous observation spaces via a Gram-determinant minimum-volume criterion, and show that continuity of the transition map over a connected state space upgrades local permutation ambiguities to a single global permutation. A small amount of labeled action data then suffices to fix this final ambiguity. These results establish demonstrator diversity as a principled source of identifiability for learning latent actions and dynamics from offline RL data.
Paper Structure (22 sections, 8 theorems, 99 equations, 1 figure)

This paper contains 22 sections, 8 theorems, 99 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Setting
  4. Observable and latent objects.
  5. Goal.
  6. Theoretical Guarantees
  7. Statewise identifiability in finite observation spaces
  8. Continuous-space identifiability via embedded Gram-determinant
  9. From state-wise to global identifiability
  10. Estimation
  11. A minimum-volume regularizer for latent transitions.
  12. Preventing policy collapse.
  13. Optional label anchoring.
  14. Final objective.
  15. Computing the embedded Gram matrix.
  16. ...and 7 more sections

Key Result

Proposition 4.1

Fix $o \in \mathcal{O}$. Suppose only the marginal next-observation law $p^*(\cdot \mid o)$ is observed, without demonstrator identity. Then in general the decomposition is not identifiable: there exist distinct pairs of latent transitions and action probabilities, not related by permutation, that induce the same observable distribution.

Figures (1)

  • Figure 1: Graphical model. Demonstrator identity $E$ affects the next observation $O_{t+1}$ only through the latent action $A_t$, while the current observation $O_t$ affects both action choice and the next observation.

Theorems & Definitions (12)

  • Proposition 4.1: Non-identifiability with a single demonstrator
  • Definition 4.1: Sufficiently scattered
  • Theorem 4.1: Statewise identifiability in finite observation spaces
  • Theorem 4.2: State-wise identifiability in continuous observation spaces
  • Theorem 4.3: From local to global permutations
  • Corollary 4.1: Global identifiability
  • Theorem A.1: Theorem 1 of fu2018identifiability
  • Lemma A.1: Column-stochasticity removes diagonal scaling
  • proof
  • Lemma A.2: Correct determinant bound
  • ...and 2 more