Deep Active Inference with Diffusion Policy and Multiple Timescale World Model for Real-World Exploration and Navigation

TL;DR

This work presents a deep active inference framework for real-world navigation that unifies exploration and goal-directed behavior by minimizing the expected free energy (EFE). It combines a diffusion policy, which generates diverse action sequences, with a multiple-timescale recurrent state-space model (MTRSSM) that provides long-horizon predictions through latent imagination. Through EFE-based evaluation of imagined trajectories, the robot selects actions that balance epistemic exploration with extrinsic goal attainment, achieving higher success rates and fewer collisions than baselines, especially in exploration-demanding scenarios. The approach demonstrates how deep generative models can scale active inference to real-world robotics, offering a principled alternative to explicit planners and highlighting directions for future work in language-guided goals and broader manipulation tasks.

Abstract

Autonomous robotic navigation in real-world environments requires exploration to acquire environmental information as well as goal-directed navigation in order to reach specified targets. Active inference (AIF) based on the free-energy principle provides a unified framework for these behaviors by minimizing the expected free energy (EFE), thereby combining epistemic and extrinsic values. To realize this practically, we propose a deep AIF framework that integrates a diffusion policy as the policy model and a multiple timescale recurrent state-space model (MTRSSM) as the world model. The diffusion policy generates diverse candidate actions while the MTRSSM predicts their long-horizon consequences through latent imagination, enabling action selection that minimizes EFE. Real-world navigation experiments demonstrated that our framework achieved higher success rates and fewer collisions compared with the baselines, particularly in exploration-demanding scenarios. These results highlight how AIF based on EFE minimization can unify exploration and goal-directed navigation in real-world robotic settings.

Figures (10)

• Figure 1: Overview of the proposed deep active inference (AIF) framework. The framework integrates a diffusion policy and a multiple timescale recurrent state-space model (MTRSSM). The diffusion policy generates diverse candidate action sequences, and the MTRSSM predicts the resulting state transitions. The expected free energy (EFE) is evaluated for each candidate sequence, and the action with the lowest EFE is selected for execution in the real-world environment.
• Figure 2: Architecture of the proposed deep AIF framework. The process comprises three steps: (1) Sample Actions: the diffusion policy generates multiple candidate action sequences of length $T_{\mathrm{F}}$, conditioned on past observations; (2) Simulate States: the MTRSSM performs latent imagination by simulating the state transitions for each candidate action sequence, sampling high- and low-level latent states in order to estimate epistemic and extrinsic values; (3) Calculate EFE: the EFE is calculated for each candidate action sequence, combining epistemic and extrinsic terms, and the action sequence with the lowest EFE is executed in the real-world environment.
• Figure 3: Computation of EFE for candidate action sequences. Each candidate action sequence $\bm{a}_t$ is simulated by the MTRSSM, using latent imagination. At each time step, the process unfolds as follows: (1) the higher-level deterministic state $d^\mathrm{h}_{\tau}$ is updated; (2) the higher-level stochastic state $s^\mathrm{h}_{\tau}$ is sampled; (3) the lower-level deterministic state $d^\mathrm{l}_{\tau}$ is updated; (4) the lower-level prior $q_\phi(s^\mathrm{l}_\tau \mid d^\mathrm{l}_\tau)$ is predicted and the stochastic state $\hat{s}^\mathrm{l}_{\tau}$ is sampled; (5) a predicted observation $\hat{o}_\tau$ is generated; and (6) the lower-level stochastic posterior $q_\phi(s^\mathrm{l}_\tau \mid d^\mathrm{l}_\tau, \hat{o}_\tau)$ is inferred using $\hat{o}_\tau$. At the lower level, the epistemic value is computed as the KL divergence between the posterior and the prior, while the extrinsic value is computed as the feature-space distance between the predicted observation $\hat{o}\tau$ and the goal observation $o_\mathrm{g}$. Combining these two terms yields the EFE $\mathcal{G}_\tau(\bm{a}_t)$, which is used to select the action sequence that balances exploration and goal-directed navigation.
• Figure 4: Experimental environment. (a) Overhead view of the indoor room. (b) Top--down map of the environment. (c) Representative observations at nine designated location--orientation patterns (1)--(9) on the map. In the real-world navigation tasks, the initial position--orientation pairs and goal images were selected from among these patterns, resulting in 18 experimental cases.
• Figure 5: Representative action sequences generated by the diffusion policy in three scenarios: (a) clear path, (b) obstacle ahead, and (c) approaching a corner. The policy adapts its action proposals to each situation and generates diverse behaviors such as forward movement and turns, illustrating its flexibility in handling different environmental contexts.