An Active Inference Model of Mouse Point-and-Click Behaviour

TL;DR

This paper addresses fine-grained pointing and click actions under perceptual delay by proposing a continuous Active Inference framework for 1D mouse pointing. The approach combines a Second-Order Lag arm and a First-Order Lag finger with observation noise, a probabilistic belief update via an Unscented Kalman Filter, and action selection by minimizing Expected Free Energy. Key findings show plausible cursor trajectories and end-point variability similar to humans, and a Fitts' Law pattern with MT = a + b log2(1 + D/W) where the agent matches the slope b but exhibits a smaller intercept a. This work demonstrates the viability of Active Inference as a continuous, probabilistic user model for HCI and outlines challenges in parameter inference, adaptation to richer biomechanics, and online learning.

Abstract

We explore the use of Active Inference (AIF) as a computational user model for spatial pointing, a key problem in Human-Computer Interaction (HCI). We present an AIF agent with continuous state, action, and observation spaces, performing one-dimensional mouse pointing and clicking. We use a simple underlying dynamic system to model the mouse cursor dynamics with realistic perceptual delay. In contrast to previous optimal feedback control-based models, the agent's actions are selected by minimizing Expected Free Energy, solely based on preference distributions over percepts, such as observing clicking a button correctly. Our results show that the agent creates plausible pointing movements and clicks when the cursor is over the target, with similar end-point variance to human users. In contrast to other models of pointing, we incorporate fully probabilistic, predictive delay compensation into the agent. The agent shows distinct behaviour for differing target difficulties without the need to retune system parameters, as done in other approaches. We discuss the simulation results and emphasize the challenges in identifying the correct configuration of an AIF agent interacting with continuous systems.

Figures (7)

• Figure 1: Mouse cursor trajectories (green lines) and clicks (green crosses) of one human user pointing and clicking at targets (red) with different distance (850px, 537px, 225px) and different widths (left: 100px, right: 20px). $t=0$ indicates the moment of a correct click. The error bars show the end-point standard deviations.
• Figure 2: A block diagram showing the interaction loop of the agent (upper) and the environment (lower). The agent includes a perceptual model, predictive delay compensation and min-EFE action planning. The simulated environment consists of a first and second-order lag modelling cursor and finger dynamics, along with click handling logic. See \ref{['alg:algorithm']} for full details.
• Figure 3: Mouse cursor and button displacement for one trial of the AIF agent from the start at 900 to $T=1750$, $W =60$px. The plot shows the real state $s(t)$ (blue), the agent's belief about $s(t-\tau)$ at time $t$ (purple) and the prediction of $s(t)$ used for planning (green). The ribbon displays +/- 3 standard deviations.
• Figure 4: Following Fitts' Law, human pointing movements show a linear relationship between movement time and the index of difficulty (ID). The simulation with AIF shows a similar trend. The data is fitted with a linear regression model. Simulation: $a = 0.22, b = 0.09, r^2 = 0.51$; User: $a = 0.46, b = 0.09, r^2 = 0.41$.
• Figure 5: Cursor trajectories for the AIF agent (upper row, blue) and a human user (lower row, green) for different target sizes (from left to right: 100px, 60px, 20px). The crosses mark mouse clicks, the red areas display the targets. $t=0$ indicates the moment of a correct click. The error bars show the end-point standard deviations. The averages for the different target sizes are: Agent: $(17.63, 10.08, 4.74)$, User: $(19.48, 13.02, 4.81)$.