Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization

TL;DR

ActionVLM addresses modality bias in vision-language temporal action localization by preserving visual grounding as the primary signal and adaptively injecting language via a lightweight debiasing module and residual aggregation. It introduces Language Advantage to estimate when language provides meaningful gains and uses alternating epochs to compute this signal efficiently. The framework combines language priors as a refinement rather than the driver, with an ActionFormer-based detection head, achieving state-of-the-art mAP on THUMOS14 and competitive results on ActivityNet-1.3 while maintaining practicality in terms of latency. Extensive analyses show that adaptive, language-aware fusion improves temporal reasoning in visually ambiguous cases and reduces overreliance on linguistic priors, making multimodal TAL more robust and scalable. The work also provides insights into the differing impacts of QA-based versus feature-aggregation formulations for modality bias in TAL.

Abstract

Temporal Action Localization (TAL) requires identifying both the boundaries and categories of actions in untrimmed videos. While vision-language models (VLMs) offer rich semantics to complement visual evidence, existing approaches tend to overemphasize linguistic priors at the expense of visual performance, leading to a pronounced modality bias. We propose ActionVLM, a vision-language aggregation framework that systematically mitigates modality bias in TAL. Our key insight is to preserve vision as the dominant signal while adaptively exploiting language only when beneficial. To this end, we introduce (i) a debiasing reweighting module that estimates the language advantage-the incremental benefit of language over vision-only predictions-and dynamically reweights language modality accordingly, and (ii) a residual aggregation strategy that treats language as a complementary refinement rather than the primary driver. This combination alleviates modality bias, reduces overconfidence from linguistic priors, and strengthens temporal reasoning. Experiments on THUMOS14 show that our model outperforms state-of-the-art by up to 3.2% mAP.

Figures (6)

• Figure 1: Visually similar actions are hard to distinguish by pixels alone, but language description provides complementary semantics that resolve the ambiguity.
• Figure 2: (a) Existing multi-modal TAL models liberatori2024testlin2022frozen distribute uniform weights to vision and language. (b) Language-centric VLMs predict in text. (c) Our proposed ActionVLM mitigates modality bias by adaptively reweighting the language modality and preserving the vision fidelity during aggregation.
• Figure 3: The overview of our ActionVLM framework. (a) Language Feature Generation: employ the language model to distill rich contextual information from action descriptions. (b) Debiasing Feature Aggregation: scale the language features by estimating the contribution, then refine the vision features with reweighted language features. (c) Proposal Generation: predict action proposals from the aggregated features.
• Figure 4: Visualization of modality bias in localizing two challenging actions, Billiard (left) and FrisbeeCatch (right). We compare our ActionVLM with a question-answering-based model (TimeMarker) and a vision-only model (AdaTAD). Numbers inside the bars denote the corresponding frame indices.
• Figure 5: Qualitative visualization under visual ambiguity. From top to bottom: (1) video frames, (2) ground truth (GT), (3) predicted boundaries with confidence (classification score from detection head), and (4) corresponding action descriptions. True positives are shown in green, while visually similar false positives without GT are highlighted in brown. Lower confidence and shorter spans indicate better calibration on ambiguous clips.