Behavior-Inspired Neural Networks for Relational Inference

TL;DR

BINN introduces a behavior-driven, graph-based framework that learns agent preferences over latent categories from observations and evolves them with nonlinear opinion dynamics to infer inter-agent relations and predict long-horizon trajectories. By embedding a nonlinear opinion dynamics inductive bias, BINN yields interpretable latent spaces, can identify mutually exclusive category structures, and outperforms several relational inference baselines on diverse mechanical and human-behavior datasets. The approach unifies expressive relational modeling with interpretability, enabling both prediction and controllable influence over agent behavior. Overall, BINN advances relational inference by linking latent preference dynamics to observable motion, with practical implications for understanding and steering multi-agent systems.

Abstract

From pedestrians to Kuramoto oscillators, interactions between agents govern how dynamical systems evolve in space and time. Discovering how these agents relate to each other has the potential to improve our understanding of the often complex dynamics that underlie these systems. Recent works learn to categorize relationships between agents based on observations of their physical behavior. These approaches model relationship categories as outcomes of a categorical distribution which is limiting and contrary to real-world systems, where relationship categories often intermingle and interact. In this work, we introduce a level of abstraction between the observable behavior of agents and the latent categories that determine their behavior. To do this, we learn a mapping from agent observations to agent preferences for a set of latent categories. The learned preferences and inter-agent proximity are integrated in a nonlinear opinion dynamics model, which allows us to naturally identify mutually exclusive categories, predict an agent's evolution in time, and control an agent's behavior. Through extensive experiments, we demonstrate the utility of our model for learning interpretable categories, and the efficacy of our model for long-horizon trajectory prediction.

Figures (11)

• Figure 1: Behavior-inspired neural network for relational inference. BINNs allow for relational inference using inter-agent distance and a learned representation of inter-category interactions. The physical states of agents are encoded to preferences for a set of latent categories. Preferences are propagated forward in time using a nonlinear opinion dynamics model with learned parameters. Predicted preferences are then decoded to physical states as predicted future physical states.
• Figure 2: Sensitivity to environmental inputs $b_{ij}$. Solid lines represent stable equilibria and dotted line represents unstable equilibria. (a) We show the pitchfork bifurcation characteristic to Equation \ref{['eqn:nonlinear_opinion_dynamics']}. The number, location, and stability of equilibria changes with the attention parameter $u_{i}$. (b) For attention $u_{i}< u^{*}$ preferences change linearly with environmental inputs. (c) For attention $u_{i}> u^{*}$ preferences change rapidly in response to environmental inputs with hysteresis encoding memory of previous states.
• Figure 3: Behavior-inspired neural network architecture overview. Our network takes agent states as inputs and outputs predicted next states, while maintaining a representation of agent preferences for a set of latent categories. The network $E_{z}$ encodes physical states to preferences for a set of latent categories and $E_{b}$ encodes physical states into latent environmental inputs. In the latent space, we compute future preferences using the nonlinear opinion dynamics block $f_{\mathrm{NOD}}$, and use the decoder $D_{x}$ to map predicted preferences to predicted physical states. The latent dynamics are unrolled for multi-step trajectory prediction.
• Figure 4: Mutually exclusive pendulum preferences. We show the observed, and latent representations of the pendulum bob. (Top-bottom) The observed position and velocity of the pendulum bob; the learned representation of agent preferences on a 2-dimensional latent space, and the learned representation of agent preferences on a 1-dimensional latent space. In the 2-dimensional space, preferences are out of phase indicating that they are mutually exclusive.
• Figure 5: Pendulum preference bifurcation diagram.(a) We show a pendulum trajectory with clockwise motion (blue) and counterclockwise motion (orange). (b) The initial preferences are near neutral, but large environmental inputs drive preferences in opposite directions resulting in opposite motions in the physical space.