Toward Learning POMDPs Beyond Full-Rank Actions and State Observability
Seiji Shaw, Travis Manderson, Chad Kessens, Nicholas Roy
TL;DR
This work develops a framework to learn discrete POMDP parameters from action-observation sequences by bridging Predictive State Representations with tensor-decomposition methods. By factorizing the Hankel matrix and recovering a similarity transform, the method obtains explicit transition and observation likelihoods up to a partition of states, enabling standard planning and reward specification. It demonstrates accurate recovery of partition-level transitions and observations and shows planning performance comparable to PSRs, while enabling reward-directed behavior from learned models. The approach broadens the applicability of spectral POMDP learning to systems with hidden states and partial observability, offering practical model-based planning benefits and a path toward scalable, structured POMDP identification.
Abstract
We are interested in enabling autonomous agents to learn and reason about systems with hidden states, such as furniture with hidden locking mechanisms. We cast this problem as learning the parameters of a discrete Partially Observable Markov Decision Process (POMDP). The agent begins with knowledge of the POMDP's actions and observation spaces, but not its state space, transitions, or observation models. These properties must be constructed from action-observation sequences. Spectral approaches to learning models of partially observable domains, such as learning Predictive State Representations (PSRs), are known to directly estimate the number of hidden states. These methods cannot, however, yield direct estimates of transition and observation likelihoods, which are important for many downstream reasoning tasks. Other approaches leverage tensor decompositions to estimate transition and observation likelihoods but often assume full state observability and full-rank transition matrices for all actions. To relax these assumptions, we study how PSRs learn transition and observation matrices up to a similarity transform, which may be estimated via tensor methods. Our method learns observation matrices and transition matrices up to a partition of states, where the states in a single partition have the same observation distributions corresponding to actions whose transition matrices are full-rank. Our experiments suggest that these partition-level transition models learned by our method, with a sufficient amount of data, meets the performance of PSRs as models to be used by standard sampling-based POMDP solvers. Furthermore, the explicit observation and transition likelihoods can be leveraged to specify planner behavior after the model has been learned.