Language Model Guided Interpretable Video Action Reasoning

TL;DR

LaIAR addresses the challenge of interpretable video action recognition by leveraging a language model to guide the training of a video model. It introduces a decoupled cross-modal framework centered on a shared DT-Former that learns from visual and semantic relations, with a threefold learning scheme: visual-semantic joint embedding, token-selection supervision, and cross-modal learning. The approach yields state-of-the-art performance on Charades and CAD-120 while providing explicit, interpretable reasoning through relational transitions mapped to semantic labels. This language-informed training paradigm enhances robustness to domain shifts and offers practical, evidence-backed explanations without requiring language inputs at inference time.

Abstract

While neural networks have excelled in video action recognition tasks, their black-box nature often obscures the understanding of their decision-making processes. Recent approaches used inherently interpretable models to analyze video actions in a manner akin to human reasoning. These models, however, usually fall short in performance compared to their black-box counterparts. In this work, we present a new framework named Language-guided Interpretable Action Recognition framework (LaIAR). LaIAR leverages knowledge from language models to enhance both the recognition capabilities and the interpretability of video models. In essence, we redefine the problem of understanding video model decisions as a task of aligning video and language models. Using the logical reasoning captured by the language model, we steer the training of the video model. This integrated approach not only improves the video model's adaptability to different domains but also boosts its overall performance. Extensive experiments on two complex video action datasets, Charades & CAD-120, validates the improved performance and interpretability of our LaIAR framework. The code of LaIAR is available at https://github.com/NingWang2049/LaIAR.

Figures (3)

• Figure 1: (a) An example of action that can be decomposed into relationship transitions (i.e., when the transition is ‘sitting on’$\rightarrow$‘not contacting’ between $<$person, bed$>$ pair, it represents the action "Someone is standing up from somewhere".). (b) Traditional two-stage methods usually predict the scene graph first, and then use language models to capture the semantic-level relationship transitions. (c) Our method exploits a language model to guide the video model to capture the relationship transition during training. During inference, our method processes videos and directly recognizes actions, providing supportive evidence.
• Figure 2: Overview of our LaIAR. The architecture comprises a language model (top) which takes the language description (represented as a spatio-temporal scene graph in ji2020action) as input and a video model (bottom) which takes the video frames as input. Both models use DT-Former to capture key relational transitions to recognize actions. We transfer knowledge across modalities using a learning scheme (i.e., Joint Embedding Space, Token Selection Supervision and Cross-Modal Learning), which can help video model benefit from language model during training. For inference, only the video model is considered.
• Figure 3: An example of action recognition performed by the proposed method and its corresponding process of providing explanations. The shaded visual relation representations indicates that it is not selected by DT-Former.