Accurate and Fast Compressed Video Captioning

TL;DR

This work tackles the inefficiency of traditional video captioning pipelines that decode frames and extract multiple features by proposing an end-to-end transformer that operates directly on compressed-domain information (I-frames, motion vectors, residuals). By leveraging a three-branch ViT encoder plus an action encoder and a multimodal decoder, the method efficiently fuses 2D context and 3D motion cues to generate captions from compressed video representations. The approach achieves competitive CIDEr scores on MSVD, MSRVTT, and VATEX while delivering substantial speedups (nearly 2x faster than the fastest prior methods). These results demonstrate the practicality of compressed-domain learning for vision-language tasks and suggest opportunities for extending the framework with additional modalities and pretraining. The work has potential impact on real-time captioning and edge deployments where decoding overhead is a bottleneck.

Abstract

Existing video captioning approaches typically require to first sample video frames from a decoded video and then conduct a subsequent process (e.g., feature extraction and/or captioning model learning). In this pipeline, manual frame sampling may ignore key information in videos and thus degrade performance. Additionally, redundant information in the sampled frames may result in low efficiency in the inference of video captioning. Addressing this, we study video captioning from a different perspective in compressed domain, which brings multi-fold advantages over the existing pipeline: 1) Compared to raw images from the decoded video, the compressed video, consisting of I-frames, motion vectors and residuals, is highly distinguishable, which allows us to leverage the entire video for learning without manual sampling through a specialized model design; 2) The captioning model is more efficient in inference as smaller and less redundant information is processed. We propose a simple yet effective end-to-end transformer in the compressed domain for video captioning that enables learning from the compressed video for captioning. We show that even with a simple design, our method can achieve state-of-the-art performance on different benchmarks while running almost 2x faster than existing approaches. Code is available at https://github.com/acherstyx/CoCap.

Figures (5)

• Figure 1: Comparing our method with prior methods for video captioning. Prior works are all based on decoding video frames. The difference between them is that some methods use offline extracted multiple features as input and generate captions, while others directly take dense video frames as input. By avoiding heavy redundant information and offline multiple feature extraction, our method speedup the caption generation process while maintaining high quality results.
• Figure 2: Comparison of model inference speed and CIDEr score on MSRVTT dataset. I, MV and Res refer to I-frame, motion vector and residual respectively. The test is run on 1 Card V100 machine with batch size set to 1.
• Figure 3: The GOP structure in compressed video. In each GOP, the first frame must be an I-frame, followed by several B/P-frames.
• Figure 4: The architecture of our proposed Compressed Video Captioner. Left: The Compressed Video Transformer which extract video representation for each GOP. A large visual backbone is used to extract visual representations from I-frame, and two small Vision Transformer is used to extract residual and motion representations from compressed domain. After that, an action encoder is used to fuse the features. Right: The Multimodal Decoder. We use a multimodal decoder with causal mask to learn caption.
• Figure 5: Qualitative results on the MSRVTT, MSVD and VATEX dataset. We show the input of our model, which is in compressed domain. The red, green and blue borders indicate I-frame, motion vector and residual, respectively.