CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization

TL;DR

CoGenAV tackles robustness in speech processing under noise and multi-speaker scenarios by learning tri-modal audio-visual-text representations through a dual objective: sequence-to-sequence contrastive synchronization and generative text prediction using a frozen ASR head. The approach preserves temporal detail in per-frame representations and achieves strong cross-modal alignment while remaining data-efficient, training on only 223 hours from LRS2. Empirically, CoGenAV delivers state-of-the-art AVSR performance (WER of 1.27 on LRS2 with clean audio) and competitive VSR, AVSS, AVSE, and ASD results, with up to 70% gains in noisy settings relative to audio-only baselines. The method's versatility and data efficiency, along with open-source intent, offer a scalable path for robust audio-visual speech processing across diverse tasks.

Abstract

The inherent synchronization between a speaker's lip movements, voice, and the underlying linguistic content offers a rich source of information for improving speech processing tasks, especially in challenging conditions where traditional audio-only systems falter. We introduce CoGenAV, a powerful and data-efficient model designed to learn versatile audio-visual representations applicable across a wide range of speech and audio-visual tasks. CoGenAV is trained by optimizing a dual objective derived from natural audio-visual synchrony, contrastive feature alignment and generative text prediction, using only 223 hours of labeled data from the LRS2 dataset. This contrastive-generative synchronization strategy effectively captures fundamental cross-modal correlations. We showcase the effectiveness and versatility of the learned CoGenAV representations on multiple benchmarks. When utilized for Audio-Visual Speech Recognition (AVSR) on LRS2, these representations contribute to achieving a state-of-the-art Word Error Rate (WER) of 1.27. They also enable strong performance in Visual Speech Recognition (VSR) with a WER of 20.5 on LRS2, and significantly improve performance in noisy environments by over 70%. Furthermore, CoGenAV representations benefit speech reconstruction tasks, boosting performance in Speech Enhancement and Separation, and achieve competitive results in audio-visual synchronization tasks like Active Speaker Detection (ASD). Our model will be open-sourced to facilitate further development and collaboration within both academia and industry.

Figures (2)

• Figure 1: Our method. Left: Audio-Visual Representation framework and the Contrastive-Generative Synchronization Training methodology. The SR Head represents the encoder and decoder components of a frozen pre-trained ASR model for Speech Recognition. Right: CoGenAV applied to diverse downstream tasks, (a) CoGenAV for AVSR; (b) CoGenAV for AVSS and AVSE; (c) CoGenAV for ASD. The blue snowflake represents weights that are frozen and non-trainable.
• Figure 2: Cross-Modal Alignment Heatmap. The brighter the color, the higher the similarity between audio-visual features, indicating better alignment. Left: $\mathcal{L}_{\text{Gen}}$ with $\mathcal{L}_{\text{Co}}$; Right: only $\mathcal{L}_{\text{Gen}}$ without $\mathcal{L}_{\text{Co}}$.