Same World, Differently Given: History-Dependent Perceptual Reorganization in Artificial Agents

Abstract

What kind of internal organization would allow an artificial agent not only to adapt its behavior, but to sustain a history-sensitive perspective on its world? I present a minimal architecture in which a slow perspective latent $g$ feeds back into perception and is itself updated through perceptual processing. This allows identical observations to be encoded differently depending on the agent's accumulated stance. The model is evaluated in a minimal gridworld with a fixed spatial scaffold and sensory perturbations. Across analyses, three results emerge: first, perturbation history leaves measurable residue in adaptive plasticity after nominal conditions are restored. Second, the perspective latent reorganizes perceptual encoding, such that identical observations are represented differently depending on prior experience. Third, only adaptive self-modulation yields the characteristic growth-then-stabilization dynamic, unlike rigid or always-open update regimes. Gross behavior remains stable throughout, suggesting that the dominant reorganization is perceptual rather than behavioral. Together, these findings identify a minimal mechanism for history-dependent perspectival organization in artificial agents.

Figures (5)

• Figure 1: Schematic overview of the extended agent architecture. Gray, green, purple, and orange denote the environment, perceptual encoding, perspective dynamics, and action pathway. The main additions are the feedback links between perception and perspective: salience gating (purple arrow), in which $g_{t-1}$ modulates perceptual encoding through the FiLM gate, and self-modulating plasticity (green arrow), in which gated perception influences the plasticity rate $\alpha_t$. Together with the update path $g_{t-1} \rightarrow \alpha_t \rightarrow g_t$, these implement the central loop of the model: perspective shapes perception, and perception regulates perspective. The updated $g_t$ then biases the fast action pathway through $s_t$. Thin black arrows indicate the main forward flow; gray arrows indicate auxiliary or implicit connections. For clarity, the observation decoder, prediction-error features, and explicit proprioceptive pathway are omitted.
• Figure 2: Gridworld environment visualization. This Pygame-based simulator renders a fixed $23 \times 7$ gridworld with a left-to-right observation noise gradient: left-side cells are noisier and right-side cells are more reliable. For analysis, the 23 columns are divided into five reporting zones (vertical white lines). Background color indicates noise level, from red (high) to teal (low). The agent (white circle) starts at the grid center. The top overlay shows timestep, position, reporting zone, action, perturbation status and trace, number of scheduled perturbations, and local observation noise $\sigma$. The right panel shows the current 8-neighbor local observation patch $x_t$. This visualization is illustrative only; all simulations were run headless.
• Figure 3: Plasticity residue under matched three-block histories. Mean adaptive plasticity $\alpha$ is compared between the first and second no-perturbation blocks across three schedules matched in total duration: Baseline ($n_P = 0\!\to\!0\!\to\!0$), Mixed perturbation ($n_P = 0\!\to\!4\!\to\!0$), and Persistent perturbation ($n_P = 4\!\to\!4\!\to\!4$). Bars show hierarchical medians, points show individual seed run values, and error bars indicate IQR. All three schedules show some decrease in $\alpha$ across blocks, but the reduction is smallest in the Baseline condition (0.284 to 0.278), largest in the Mixed condition (0.278 to 0.228), and intermediate in the Persistent condition (0.293 to 0.263).
• Figure 4: Perspective reorganizes perception of the same input. The comparison perspective $g$ was obtained from a mixed perturbation run ($n_P = 0\!\to\!4\!\to\!0$). (a) PCA projection of probe encodings under the late post-perturbation perspective state $g_2$ and the null condition $g\!=\!0$, shown with covariance ellipses. Cluster centroids are marked with crosses. (b) Signed per-dimension difference $z_t(g_2) - z_t(g\!=\!0)$ averaged across probe observations. (c) Per-dimension FiLM salience-gating coefficients $\gamma$ in Block 0 (pre-perturbation, $n_P\!=\!0$) and Block 2 (post-perturbation, $n_P\!=\!0$) of the mixed-history run.
• Figure 5: Ablation analysis of self-modulating plasticity.(a) Episode-wise adaptive plasticity $\alpha$ with perturbation ($n_P=4$) and without perturbation ($n_P=0$). In both conditions, $\alpha$ rises rapidly early in training, but only the perturbed condition later declines below baseline. (b) Perspective magnitude $\|g\|$ across four update regimes: Adaptive with perturbation, Adaptive baseline, Rigid ($\alpha=0.05$), and Open ($\alpha=0.80$). The rigid regime shows continued growth, the open regime remains relatively flat, and the adaptive regimes lie between these extremes, with slightly larger late-stage $\|g\|$ under perturbation.