Detecting Perspective Shifts in Multi-agent Systems

TL;DR

This work addresses the challenge of detecting perspective shifts in black-box multi-agent systems by introducing the Temporal Data Kernel Perspective Space (TDKPS), a time-aware, low-dimensional embedding of agents. It couples TDkPS with principled agent- and group-level permutation-based tests (including energy-distance and distance-correlation based methods) to detect temporal changes in behavior without requiring access to internal mechanisms. Through simulations and a natural experiment using digital congresspersons and COVID-era public-health queries, the authors demonstrate near-optimal power, robust performance, and substantial computational efficiency for group-level inference. The approach provides a scalable framework for monitoring dynamic agent ecosystems as generative agents continue to proliferate, with implications for safety, reliability, and interpretability in real-world deployments.

Abstract

Generative models augmented with external tools and update mechanisms (or \textit{agents}) have demonstrated capabilities beyond intelligent prompting of base models. As agent use proliferates, dynamic multi-agent systems have naturally emerged. Recent work has investigated the theoretical and empirical properties of low-dimensional representations of agents based on query responses at a single time point. This paper introduces the Temporal Data Kernel Perspective Space (TDKPS), which jointly embeds agents across time, and proposes several novel hypothesis tests for detecting behavioral change at the agent- and group-level in black-box multi-agent systems. We characterize the empirical properties of our proposed tests, including their sensitivity to key hyperparameters, in simulations motivated by a multi-agent system of evolving digital personas. Finally, we demonstrate via natural experiment that our proposed tests detect changes that correlate sensitively, specifically, and significantly with a real exogenous event. As far as we are aware, TDKPS is the first principled framework for monitoring behavioral dynamics in black-box multi-agent systems -- a critical capability as generative agent deployment continues to scale.

Figures (6)

• Figure 1: The $T=2$, 2-d Temporal Data Kernel Perspective Space ("TDKPS") of a multi-agent system consisting of generative agents parameterized by different, dynamic retrieval datasets. Each dot/triangle is an agent. TDKPS enables interpretable and principled analysis of multi-agent systems in the black-box setting. For more experimental details, see Section \ref{['sec:data']}.
• Figure 2: Simulation design and TDKPS schematic. Illustration with $N = 50$ agents, $p = 10$ total dimensions, $p_s = 3$ signal dimensions, and effect size $\tau = 1$. (A) Class-specific temporal dynamics: Class 0 (blue) exhibits temporal change transitioning from front-loaded ($t = 1$) to back-loaded ($t = 2$). Class 1 (orange) shows no temporal change. (B) True latent structure in the first two dimensions, with lines connecting each agent across timepoints. (C) Observed query responses result from random orthogonal transformations of the latent space (three example queries shown) with added measurement noise, obscuring the latent structure. Illustrated is a random rotation, which is a type of orthogonal transformation. (D) TDKPS embedding recovers the relational structure from (B).
• Figure 3: Four simulations demonstrate strengths of TDKPS. Power curves show mean rejection rates across 50 trials. Shaded regions: 95% confidence intervals; horizontal dotted line: nominal $\alpha = 0.05$. (A) Power increases with effect size. (B) More agents improve embedding stability (only TDKPS varies; other methods ignore pairwise structure). (C) Additional queries improve power. (D) Replicates substantially impact TDKPS power and validity.
• Figure 4: Agent-level behavior shift after COVID-19 onset most pronounced for public health queries.(I) TDKPS embeddings projected via LDA to separate Republicans (red) and Democrats (blue); faint lines show individual trajectories, bold lines show party averages. Yellow shading: first two years post-COVID-19 onset. (II) Empirical CDFs of $p$-values from agent-level tests, grouped by COVID-19 proximity (yellow/black). Deviation from diagonal indicates temporal shifts. (III) Rank of average TDKPS distance vs. temporal distance from April 2021. (A) Public health queries ($K = 0.51$, $p = 0.014$), (B) general political queries ($K = 0.34$, $p = 0.11$), (C) orthogonal queries ($K = 0.01$, $p = 0.956$).
• Figure 5: Simulations demonstrate strengths of PE$\circ$TDKPS. Power curves show mean rejection rates across 200 trials (40 for D). Shaded regions: 95% confidence intervals; horizontal dotted line: $\alpha = 0.05$. Power increases with (A) effect size, (B) agents, (C) queries, and (D) replicates for PE$\circ$TDKPS. Type I error of PE$\circ$TDKPS remains $\approx \alpha$ across all conditions.