Perceptual Motor Learning with Active Inference Framework for Robust Lateral Control

TL;DR

This work introduces Perceptual Motor Learning (PML) integrated with an Active Inference Framework (AIF) to enhance lateral control in Highly Automated Vehicles, emphasizing vision-only perception and a learned internal world model to generalize across diverse driving environments without extensive retraining. The method combines a forward transition model trained offline with an action-selection mechanism that minimizes Expected Free Energy (EFE), using a SSIM-based distance to a preferred future state for lane-keeping. Experimental validation in CARLA demonstrates superior adaptability and competitive performance against imitation learning and benchmark methods, with clear generalization benefits from not training on target towns. The results highlight the potential of PML-driven active inference as a scalable, data-efficient approach for robust autonomous driving in real-world, variable environments.

Abstract

This paper presents a novel Perceptual Motor Learning (PML) framework integrated with Active Inference (AIF) to enhance lateral control in Highly Automated Vehicles (HAVs). PML, inspired by human motor learning, emphasizes the seamless integration of perception and action, enabling efficient decision-making in dynamic environments. Traditional autonomous driving approaches--including modular pipelines, imitation learning, and reinforcement learning--struggle with adaptability, generalization, and computational efficiency. In contrast, PML with AIF leverages a generative model to minimize prediction error ("surprise") and actively shape vehicle control based on learned perceptual-motor representations. Our approach unifies deep learning with active inference principles, allowing HAVs to perform lane-keeping maneuvers with minimal data and without extensive retraining across different environments. Extensive experiments in the CARLA simulator demonstrate that PML with AIF enhances adaptability without increasing computational overhead while achieving performance comparable to conventional methods. These findings highlight the potential of PML-driven active inference as a robust alternative for real-world autonomous driving applications.

Figures (6)

• Figure 1: Overview of Perceptual Motor Learning (PML) with Active Inference Frameworks (AIF). The agent senses the environment through its sensors and makes observations. These observations shape the agent's beliefs and perception of the world. Based on its perception, the agent takes actions that, in turn, affect the real environment. (Icon made by Freepik from www.flaticon.com)
• Figure 2: Task-agnostic offline perception model training. A hidden state $s^*_t$ is perceived through the observation model $f_o$. An observation $o_t$ is encoded and kept inside the agent as a perceived state. When an action $a_t$ is applied to the environment by the agent, the causal changes in the environment should be observed as $\hat{o}_{t+1}$. The forward transition model $f_s$ is being trained using the distance model $f_d$ with $o_{t+1}$ and $\hat{o}_{t+1}$. The encoder and decoder can be the inside of $f_s$ in practice.
• Figure 3: Action selection. The forward transition model $f_s$ generates causal observations based on covert actions. $o^c_t$ is preference for a task. The distance model $f_d$ calculates the dissimilarity between the observations and the preference and finds which action will generate the most similar observations to the preference.
• Figure 4: Predicted and the ground truth image for semantic segmented images with the SSIM difference between them.
• Figure 5: Tracks used for testing in Town01, Town04, and Town06 from left to right. [A] shows the straight, [B] shows the one-turn, and [C] shows the two-turn track.