CBIL: Collective Behavior Imitation Learning for Fish from Real Videos

TL;DR

CBIL tackles the challenge of learning realistic fish schooling directly from 2D videos without relying on 3D motion capture. It introduces a Masked Video AutoEncoder (MVAE) to extract latent implicit states from videos and uses a GAIL-based imitation framework augmented with implicit-state clustering and bio-inspired rewards to match video distributions in the latent space. The approach supports synthesizing multiple patterns (circling, alignment, aggregation, feeding, chasing) across species and enables abnormal-behavior detection in the wild, outperforming prior methods on cross-view and distribution-based metrics. By combining data-driven priors with biologically grounded regularization, CBIL achieves diverse, robust, and transferable collective motion synthesis with practical animation and analysis benefits.

Abstract

Reproducing realistic collective behaviors presents a captivating yet formidable challenge. Traditional rule-based methods rely on hand-crafted principles, limiting motion diversity and realism in generated collective behaviors. Recent imitation learning methods learn from data but often require ground truth motion trajectories and struggle with authenticity, especially in high-density groups with erratic movements. In this paper, we present a scalable approach, Collective Behavior Imitation Learning (CBIL), for learning fish schooling behavior directly from videos, without relying on captured motion trajectories. Our method first leverages Video Representation Learning, where a Masked Video AutoEncoder (MVAE) extracts implicit states from video inputs in a self-supervised manner. The MVAE effectively maps 2D observations to implicit states that are compact and expressive for following the imitation learning stage. Then, we propose a novel adversarial imitation learning method to effectively capture complex movements of the schools of fish, allowing for efficient imitation of the distribution for motion patterns measured in the latent space. It also incorporates bio-inspired rewards alongside priors to regularize and stabilize training. Once trained, CBIL can be used for various animation tasks with the learned collective motion priors. We further show its effectiveness across different species. Finally, we demonstrate the application of our system in detecting abnormal fish behavior from in-the-wild videos.

Figures (16)

• Figure 1: Diverse Fish Behaviors.
• Figure 2: An overview of CBIL. Our framework has three stages: the visual representation learning stage, the collective behavior imitation learning stage, and the crowd animation stage for various animation tasks. In the first stage, we train the MVAE to learn mappings from video inputs to latent states. These latent states are later used in our collective behavior imitation learning. In the second stage, we employ both data-driven motion prior learned from videos and bio-inspired motion prior for imitation learning. Finally, we demonstrate that CBIL is applicable to diverse fish school animations such as circling, alignment, and aggregation.
• Figure 3: Masked Video AutoEncoder. We use reference and rendered videos of simulated fish schools to train the model. Here, $z_t$ denotes low-dimensional implicit states with a dimensionality of 100, and $\hat{o}_t$ denotes reconstructed clips.
• Figure 4: Illustration of some of the 3D simulated fish models we utilize. Our scalable approach enables the training of policies for a broad spectrum of fish species.
• Figure 5: t-SNE visualization of MVAE latent representations for reference and simulated circling videos during pre-training. Top: Clockwise, Bottom: Counterclockwise. Colors represent different sources: blue for reference, red for rule-based generated, and green for randomization.