OMCAT: Omni Context Aware Transformer

TL;DR

The model demonstrates state-of-the-art performance on Audio-Visual Question Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in temporal reasoning and cross-modal alignment, as validated through comprehensive experiments and ablation studies.

Abstract

Large Language Models (LLMs) have made significant strides in text generation and comprehension, with recent advancements extending into multimodal LLMs that integrate visual and audio inputs. However, these models continue to struggle with fine-grained, cross-modal temporal understanding, particularly when correlating events across audio and video streams. We address these challenges with two key contributions: a new dataset and model, called OCTAV and OMCAT respectively. OCTAV (Omni Context and Temporal Audio Video) is a novel dataset designed to capture event transitions across audio and video. Second, OMCAT (Omni Context Aware Transformer) is a powerful model that leverages RoTE (Rotary Time Embeddings), an innovative extension of RoPE, to enhance temporal grounding and computational efficiency in time-anchored tasks. Through a robust three-stage training pipeline-feature alignment, instruction tuning, and OCTAV-specific training-OMCAT excels in cross-modal temporal understanding. Our model demonstrates state-of-the-art performance on Audio-Visual Question Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in temporal reasoning and cross-modal alignment, as validated through comprehensive experiments and ablation studies. Our dataset and code will be made publicly available. The link to our demo page is https://om-cat.github.io.

Figures (7)

• Figure 1: Illustration of a video sequence from our proposed OCTAV dataset. The annotations highlight key moments, including the timing of the audio and visual events.
• Figure 2: Overview of the OMCAT pipeline. Video frames are processed through a frozen visual encoder, while audio frames are encoded using a frozen audio encoder. Extracted features are fine-tuned through adaptor layers across all three stages. The LLM remains frozen in Stage 1 and is fine-tuned in Stages 2 and 3. The purple blocks represent time alignment modules, with only one of them activated during training. $\angle$ in bottom right denotes the rotation angle.
• Figure 3: Qualitative comparison of OMCAT with GroundingGPT on the OCTAV-MT dataset.
• Figure 4: Question-answer pairs from the proposed OCTAV-ST dataset.
• Figure 5: Multi-turn dialogue examples from the ActivityNet subset of our OCTAV-MT dataset.