OmniVid: A Generative Framework for Universal Video Understanding

TL;DR

This work seeks to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens, which enables a variety of video tasks to be formulated as video-grounded token generation.

Abstract

The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution. Despite sharing a common goal, different tasks often rely on distinct model architectures and annotation formats. In contrast, natural language processing benefits from a unified output space, i.e., text sequences, which simplifies the training of powerful foundational language models, such as GPT-3, with extensive training corpora. Inspired by this, we seek to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens. In this way, a variety of video tasks could be formulated as video-grounded token generation. This enables us to address various types of video tasks, including classification (such as action recognition), captioning (covering clip captioning, video question answering, and dense video captioning), and localization tasks (such as visual object tracking) within a fully shared encoder-decoder architecture, following a generative framework. Through comprehensive experiments, we demonstrate such a simple and straightforward idea is quite effective and can achieve state-of-the-art or competitive results on seven video benchmarks, providing a novel perspective for more universal video understanding. Code is available at https://github.com/wangjk666/OmniVid.

Figures (6)

• Figure 1: A conceptual comparison between existing video models and OmniViD.
• Figure 2: Illustration of the time tokens and box tokens in OmniViD.
• Figure 3: Input $\&$ output of different video tasks. S/B/W/T: Sentence / Box / Word / Time, Pro. / Tok.: Prompt / Token.
• Figure 4: Architecture of OmniViD. The Mixed Q-former aggregates the frame features into three types of queries, i.e., content queries, text queries, and box queries. After that, the queries obtained from different frames are input to a temporal encoder for temporal modeling. Finally, the token decoder generates a sequence of tokens conditioned on the multimodal inputs.
• Figure 5: Comparison between joint and separate training.