Concepts in Motion: Temporal Bottlenecks for Interpretable Video Classification

TL;DR

A transformer-based concept architecture that operates on sequences of temporally grounded concept activations, by employing per-concept temporal self-attention to model when individual concepts recur and how their temporal patterns contribute to predictions.

Abstract

Concept Bottleneck Models (CBMs) enable interpretable image classification by structuring predictions around human-understandable concepts, but extending this paradigm to video remains challenging due to the difficulty of extracting concepts and modeling them over time. In this paper, we introduce $\textbf{MoTIF}$ (Moving Temporal Interpretable Framework), a transformer-based concept architecture that operates on sequences of temporally grounded concept activations, by employing per-concept temporal self-attention to model when individual concepts recur and how their temporal patterns contribute to predictions. Central to the framework is an agentic concept discovery module to automatically extract object- and action-centric textual concepts from videos, yielding temporally expressive concept sets without manual supervision. Across multiple video benchmarks, this combination substantially narrows the performance gap between interpretable and black-box video models while maintaining faithful and temporally grounded concept explanations. Code available at $\href{https://github.com/patrick-knab/MoTIF}{github.com/patrick-knab/MoTIF}$.

Figures (14)

• Figure 1: Overview of MoTIF. Video classification pipeline with agentic concept discovery and its three explanation modes.
• Figure 2: MoTIF. The framework takes videos as input and produces local concept explanations for local windows, global explanations for entire videos, and temporal dependency maps from the attention heads of the transformer module. Model represents MoTIF (ViT-L14) and sample frames are from HMDB51 hmdb, licensed under CC BY 4.0.
• Figure 3: Effect of log-sum-exp temperature $\tau$ on accuracy and entropy. Accuracy remains stable, while concept- and logit-level entropy decrease as $\tau$ increases.
• Figure 4: Full vs. diagonal attention. Train and test accuracy with and without enforcing diagonal attention over five seeds.
• Figure 5: MoTIF explanations. Example videos from Breakfast and UCF101 with correct classifications, illustrating the three explanation modes supported by MoTIF (ViT-L14).