Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition

TL;DR

DANCE introduces a disentangled, concept-based framework for video action recognition that grounds predictions in three interpretable concept types: motion dynamics (pose sequences), objects, and scenes. Built as an ante-hoc concept bottleneck, DANCE enforces prediction through these concepts, using pose clustering for motion dynamics and LLM-driven, pseudo-labeled object/scene concepts via a vision-language dual encoder. Empirical results on KTH, Penn Action, HAA500, and UCF-101 show improved explanation clarity with competitive predictive performance, supported by a user study and qualitative analyses. The approach also enables practical model debugging and editing, including domain-shift recovery, without retraining. Overall, DANCE demonstrates that structured, motion-aware explanations can be faithful, interpretable, and actionable for video understanding tasks.

Abstract

Effective explanations of video action recognition models should disentangle how movements unfold over time from the surrounding spatial context. However, existing methods based on saliency produce entangled explanations, making it unclear whether predictions rely on motion or spatial context. Language-based approaches offer structure but often fail to explain motions due to their tacit nature -- intuitively understood but difficult to verbalize. To address these challenges, we propose Disentangled Action aNd Context concept-based Explainable (DANCE) video action recognition, a framework that predicts actions through disentangled concept types: motion dynamics, objects, and scenes. We define motion dynamics concepts as human pose sequences. We employ a large language model to automatically extract object and scene concepts. Built on an ante-hoc concept bottleneck design, DANCE enforces prediction through these concepts. Experiments on four datasets -- KTH, Penn Action, HAA500, and UCF-101 -- demonstrate that DANCE significantly improves explanation clarity with competitive performance. We validate the superior interpretability of DANCE through a user study. Experimental results also show that DANCE is beneficial for model debugging, editing, and failure analysis.

Figures (10)

• Figure 1: Disentangled concepts speak louder than words. Spatio-temporal attribution methods provide unstructured explanations that are often ambiguous to human users. Given a Baseball Swing video, (a) visual explanations from 3D-saliency stergiou2019saliency and VTCD vtcd fail to clarify whether the prediction is driven by motion (e.g., torso twist), objects (e.g., jersey/helmet), or scene context (e.g., baseball field). (b) Language-based approaches offer more structure but remain ambiguous for motion, as it is tacit knowledge—intuitively understood but hard to verbalize. Verbal descriptions of motion often lack clarity and are difficult to interpret. (c) In contrast, DANCEdisentangles motion and context to provide structured explanations. Pose sequences capture motion dynamics in an intuitive, appearance-invariant form, while we clearly convey object and scene concepts via text.
• Figure 2: Overview of DANCE. Given an input video, DANCE first extracts video features using a pretrained video backbone encoder. Then, three disentangled concept layers project the video features onto their own concept space-motion dynamics, object, and scene-producing disentangled activations. The interpretable classification layer linearly combines these concept activations to predict the action class. By explicitly disentangling concept types, DANCE provides structured explanations that better align with how humans perceive actions by separating motion dynamics from the spatial context.
• Figure 3: Concept discovery and labeling process of DANCE. (a) Given a training video, we extract $S$ key clips with length $L$ centered at keyframes identified by a keyframe detection algorithm. We then apply a 2D pose estimator to obtain human pose sequences from these key clips. By clustering all pose sequences across the training set, we cluster them to define each cluster as a motion dynamics concept. (b) For each video, we derive binary motion dynamics concept labels by aggregating the cluster assignment tensor across its key clips. (c) To discover object concepts, we query GPT-4o hurst2024gpt with prompts containing action class names, yielding a set of object concepts for the dataset. (d) Given a video and the object concept set, we compute concept pseudo labels using a vision-language dual encoder. Specifically, we obtain a concept pseudo label vector by multiplying the object concept embedding matrix with the video embedding vector. We can obtain scene concept labels analogously.
• Figure 4: Sample-level explanation. We compare (a) DANCE with (b) existing attribution-based methods: GradCAM gradcam attribution and Saliency Tubes stergiou2019saliency.
• Figure 5: Visualization of model weights of similar action class pairs. For each example, we show a Sankey diagram of the final layer weights associated with a similar action class pair.