Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning

TL;DR

<3-5 sentence high-level summary>PE-AV presents a scalable audio–video–text encoder family built on the Perception Encoder, extending AVT representations to audio and enabling unified embeddings across audio–video, audio–text, and video–text. A two-stage audiovisual data engine generates high-quality synthetic captions for about 100M AV pairs, complemented by real captions, to train a ten-pair cross-modal objective that strengthens alignment across modalities. The authors introduce PE-A-Frame to achieve fine-grained audio-frame–text alignment for sound event detection, demonstrating robust open- and closed-vocabulary SED performance. Across zero-shot sound, music, speech, and video benchmarks, PE-AV achieves state-of-the-art results and shows strong joint-modal capabilities, signaling a significant step toward omni-modal perception foundations.

Abstract

We introduce Perception Encoder Audiovisual, PE-AV, a new family of encoders for audio and video understanding trained with scaled contrastive learning. Built on PE, PE-AV makes several key contributions to extend representations to audio, and natively support joint embeddings across audio-video, audio-text, and video-text modalities. PE-AV's unified cross-modal embeddings enable novel tasks such as speech retrieval, and set a new state of the art across standard audio and video benchmarks. We unlock this by building a strong audiovisual data engine that synthesizes high-quality captions for O(100M) audio-video pairs, enabling large-scale supervision consistent across modalities. Our audio data includes speech, music, and general sound effects-avoiding single-domain limitations common in prior work. We exploit ten pairwise contrastive objectives, showing that scaling cross-modality and caption-type pairs strengthens alignment and improves zero-shot performance. We further develop PE-A-Frame by fine-tuning PE-AV with frame-level contrastive objectives, enabling fine-grained audio-frame-to-text alignment for tasks such as sound event detection.

Figures (14)

• Figure 1: Perception Encoder Audiovisual () is a family of audio-video-text (AVT) encoders. By scaling coverage of contrastive learning, model size, and training with synthetically aligned video-audio-text pairs across diverse domains, achieves state-of-the-art performance on a wide range of zero-shot sound, music, speech, and video classification and retrieval tasks.
• Figure 2: Perception Encoder-AudioVisual () is composed of an audio encoder, a frame encoder, video encoder, audio-video fusion encoder and a text encoder. For audio we use DAC-VAE to encode raw audio waveforms. For video we directly encode raw RGB frames. We use eight contrastive loss to associate embeddings of eight types of multimodal pairs. We use two extra pairs during fine-tuning stage adding to a total of ten loss pairs.
• Figure 3: Data Engine: Stage-1 Synthetic Captions: In the first stage we use a synthetic captioning pipeline that uses Llama 3.1 8B to combine information from weak audio captioning models together with confidence using the Joint-CLAP vyas2023audiobox and video captioner to generate audio, visual, and audio-visual captions.
• Figure 4: EnCLAP and CoNeTTE captions often provide complementary information and the confidence scores reflect the accuracy reasonably, making them favorable to combine with an LLM. Video captions provide strong context. Together this provides strong audio and visual cues for LLM rewriting.
• Figure 5: Data Engine: Stage-2 Improved Captions: We improve both the audio and visual captions in the second stage. For the visual captions, we use PLM plm to generate the video captions and an LLM to summarize the first stage and PLM captions. For audio captions, we follow the PLM recipe plm to train a PLM-AV model to generate three different audio caption variants focused on audio events, caption, and acoustic environment.