Learning Task-Agnostic Motifs to Capture the Continuous Nature of Animal Behavior

TL;DR

The paper tackles the problem that traditional behavior segmentation reduces animal movement to discrete syllables, which fails to capture the continuous, compositional nature of motor motifs. It introduces Motif-based Continuous Dynamics (MCD), a reinforcement-learning–driven framework that learns motif sets and continuously evolving motif weights to reconstruct and predict complex trajectories across tasks. By deriving motifs via spectral decomposition in the discrete setting and an energy-based model with noise-contrastive estimation in the continuous setting, MCD provides a generative, assumption-light account of how continuous motor patterns combine to produce observable behavior, including a time-varying reward interpretation for inverse RL analyses. Empirically, MCD demonstrates reusable, interpretable motifs in a gridworld, labyrinth navigation, and freely moving mice, offering improved trajectory realism and insight into the motor-building blocks underlying natural behavior.

Abstract

Animals flexibly recombine a finite set of core motor motifs to meet diverse task demands, but existing behavior segmentation methods oversimplify this process by imposing discrete syllables under restrictive generative assumptions. To better capture the continuous structure of behavior generation, we introduce motif-based continuous dynamics (MCD) discovery, a framework that (1) uncovers interpretable motif sets as latent basis functions of behavior by leveraging representations of behavioral transition structure, and (2) models behavioral dynamics as continuously evolving mixtures of these motifs. We validate MCD on a multi-task gridworld, a labyrinth navigation task, and freely moving animal behavior. Across settings, it identifies reusable motif components, captures continuous compositional dynamics, and generates realistic trajectories beyond the capabilities of traditional discrete segmentation models. By providing a generative account of how complex animal behaviors emerge from dynamic combinations of fundamental motor motifs, our approach advances the quantitative study of natural behavior.

Figures (7)

• Figure 1: A policy for a movement can be seen as a blend of "vocabularies" from a dictionary containing fundamental motor motifs.
• Figure 2: A. State-action map: each of the nine grids is divided into four cells representing action values (Up, Down, Left, Right). In task $i$, high reward is assigned to $(s,a)$ pairs moving toward the $i$th location. B. Explained variance of the top 15 principal components; variance drops near zero after PC7. C. Left: true rewards for all 9 tasks (yellow = high, green = low). Right: recovered rewards. D. Reward weight $w$ and top 8 PC motifs from the $\phi$ matrix. Reward weights indicate the contribution of the top 8 PC motifs to each task. In the PC motif plot, red indicates positive feature values, blue indicates negative.
• Figure 3: A. True and recovered rewards for the three tasks. B. Two dominant motifs for the water and home tasks respectively. Each motif indicates that taking a specific action (orange arrow) toward the dark purple state yields a high value. C.$w$ values for all tasks, colored by task; blue stars highlight high-weight motifs for water seeking, and orange for home seeking, all shown in (B).
• Figure 4: A. AUC on training and test sets. We take an example behavior video and run our algorithm. We visualize the motif weights $u(t)$ and show the representative motifs in B. For the baseline SemiSeg, we show the latent skills and segmentation results in C. For the baseline OPAL, we show the latent skills and segmentation results in D. Then we show the segmentation results of Keypoint-MoSeq weinreb2024keypoint in E and the representative motifs/syllables in F.
• Figure 5: A. State-action maps for all 64 motifs. B. Reward weight $w$ for all 64 motifs.

Theorems & Definitions (3)

• Definition 1: Motif Set
• Proposition 1
• Proposition 2: Connection to Motif Definition