Track the Answer: Extending TextVQA from Image to Video with Spatio-Temporal Clues

TL;DR

This work tackles Video TextVQA by bridging image-based TextVQA methods with video-specific challenges. It introduces TEA, a generative framework built on a T5 backbone that explicitly recovers spatio-temporal relationships among scene text and visual entities via a Temporal Convolution Module and an OCR-enhanced Relative Spatial Module, while simultaneously generating concise, question-relevant clues through a Scene Text Aware Clues Aggregation module. By leveraging a global context from CLIP and scene-text–driven cross-attention, TEA improves reasoning over long videos and reduces interference from irrelevant frames. Extensive experiments on M4-ViteVQA and RoadTextVQA demonstrate that TEA-Large achieves state-of-the-art results, outperforming image TextVQA methods, video-language pretraining, and video LLMs, with meaningful gains in both accuracy and ANLS. The approach advances practical video understanding by enabling robust reading and reasoning over dynamic scene text in real-world contexts.

Abstract

Video text-based visual question answering (Video TextVQA) is a practical task that aims to answer questions by jointly reasoning textual and visual information in a given video. Inspired by the development of TextVQA in image domain, existing Video TextVQA approaches leverage a language model (e.g. T5) to process text-rich multiple frames and generate answers auto-regressively. Nevertheless, the spatio-temporal relationships among visual entities (including scene text and objects) will be disrupted and models are susceptible to interference from unrelated information, resulting in irrational reasoning and inaccurate answering. To tackle these challenges, we propose the TEA (stands for ``\textbf{T}rack th\textbf{E} \textbf{A}nswer'') method that better extends the generative TextVQA framework from image to video. TEA recovers the spatio-temporal relationships in a complementary way and incorporates OCR-aware clues to enhance the quality of reasoning questions. Extensive experiments on several public Video TextVQA datasets validate the effectiveness and generalization of our framework. TEA outperforms existing TextVQA methods, video-language pretraining methods and video large language models by great margins.

Figures (4)

• Figure 1: An example to illustrate the challenges in Video TextVQA and the overview of our proposed method. For this given question, the Video TextVQA model requires to first analyze the spatial relationships to locate the answer instance, and then capture the temporal correspondence to track and recognize the answer content. Besides, unrelated frames may interfere with model reasoning.
• Figure 2: (a) Overall architecture of TEA. Blue and green token $S^{tok}_{i,j}$ represents the $j$th OCR and object token in $i$th frame, respectively. (b) Illustration of spatio-temporal recovery for visual entities. Temporal convolution and spatial attention bias represents the Temporal Convolution Module and OCR-enhanced relative spatial module, respectively.
• Figure 3: Illustration of scene text aware clues aggregation.
• Figure 4: (a) A case study on M4-ViteVQA. The red boxes indicate the regions related to the question. (b) Visualization of cross-attention between questions and scene text across each video frame in the answer generation process. Red boxes in the raw video frames represent the locations of the corresponding highlighted scene text.