A Model-free Biomimetics Algorithm for Deterministic Partially Observable Markov Decision Process

TL;DR

The paper addresses decision making under uncertainty in DET-POMDPs, a deterministic-state, many-to-one observation subclass of POMDPs that challenges model-free solutions. It introduces BIOMAP, a Desert-Ant inspired biomimetic algorithm that builds a Compact Vector Graph and uses a MDP-Graph-Automaton framework to transform DET-POMDPs into fully observable MDPs, enabling stable policy derivation via shortest-path methods. The work formalizes the problem, analyzes the Cognitive Fog bias through Q-value variance, and provides a Boundary Arbiter to prevent unbounded exploration, culminating in a four-phase BIOMAP pipeline with complexity guarantees. Experimental results on a Masking Cliff Walking environment show BIOMAP achieving competitive or superior performance to several model-based solvers, demonstrating robustness to environmental deception and potential for reliable deployment in DET-POMDP-prone domains.

Abstract

Partially Observable Markov Decision Process (POMDP) is a mathematical framework for modeling decision-making under uncertainty, where the agent's observations are incomplete and the underlying system dynamics are probabilistic. Solving the POMDP problem within the model-free paradigm is challenging for agents due to the inherent difficulty in accurately identifying and distinguishing between states and observations. We define such a difficult problem as a DETerministic Partially Observable Markov Decision Process (DET-POMDP) problem, which is a specific setting of POMDP. In this problem, states and observations are in a many-to-one relationship. The state is obscured, and its relationship is less apparent to the agent. This creates obstacles for the agent to infer the state through observations. To effectively address this problem, we convert DET-POMDP into a fully observable MDP using a model-free biomimetics algorithm called BIOMAP. BIOMAP is based on the MDP Graph Automaton framework to distinguish authentic environmental information from fraudulent data. Thus, it enhances the agent's ability to develop stable policies against DET-POMDP. The experimental results highlight the superior capabilities of BIOMAP in maintaining operational effectiveness and environmental reparability in the presence of environmental deceptions when compared with existing POMDP solvers. This research opens up new avenues for the deployment of reliable POMDP-based systems in fields that are particularly susceptible to DET-POMDP problems.

Figures (5)

• Figure 1: General POMDP vs. DET-POMDP example. In DET-POMDP with environment-free modeling, the correspondence between states and observations is more challenging compared to general POMDPs. This difficulty arises from the unchanging pattern of occurrence of observations, which hampers the process of establishing a reliable mapping between states and observations.
• Figure 2: Schematic Diagrams and Flowchart of the Desert Ant's Navigational Biometrics and BIOMAP. (a) is a schematic diagram of the biometrics of the desert ant's navigational ability; (b) is a schematic diagram of the desert ant's angle monitoring; (c) is a schematic diagram of the desert ant's path integration; (d) is an example of the bionic algorithm; and (e) is a flowchart of the BIOMAP algorithm.
• Figure 3: Structure of MDP-Graph-Automaton.
• Figure 4: This figure illustrates the design of the Masking Cliff Walking experiment, showcasing different settings of observability. In setting (a), the environment is fully observable, denoted by the absence of any masking. The agent has complete knowledge of the grid configuration at all times. In settings (b) to (f), the environment becomes partially observable, introducing various forms of masking to limit the agent's visibility of the grid world. Setting (b) demonstrates different masking directions; Setting (c) exhibits different masking numbers; Setting (d) shows a combination of continuous and discrete masking; Setting (e) displays different masking layers; Setting (f) presents a hybrid setting that combines elements from settings (b) to (e).
• Figure 5: Visualization of results. In (a), $\textcircled{1}$ is the process of experimental settings, $\textcircled{2} - \textcircled{6}$ are BIOMAP's working process on the masking cliff walking; (b) is a recovered Action Vector graph for Cliff Walking with masking (direction: row, number = $12$, continuity = True, layer = $3$) in (a).

Theorems & Definitions (19)

• Definition 1: DET-POMDP model bonet2012deterministic
• Definition 2: History trajectory
• Theorem 1
• Definition 3: Action Unit Vector
• Definition 4: Compact Vector Graph
• Definition 5: MDP with graph representation
• Definition 6: Dual relation
• Lemma 2
• Lemma 3
• Theorem 4