Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment

TL;DR

The paper tackles robust, label-efficient video representation learning by aligning temporal sequences with a differentiable local alignment mechanism. It introduces the Local-Alignment Contrastive (LAC) loss, which fuses a differentiable Smith-Waterman local alignment with a contrastive objective, enabling dynamic, subsequence-aware matching with learnable gap penalties. A transformer-based encoder extracts frame-wise features, and the final objective combines the local alignment loss with a Gaussian-weighted contrastive term, improving action recognition benchmarks on Pouring and PennAction. This approach advances self-supervised video understanding by emphasizing local temporal structure and discriminative embedding learning, with practical impact on downstream action analysis tasks. All mathematical formulations are presented in $...$ to ensure precise, machine-readable representation.

Abstract

Robust frame-wise embeddings are essential to perform video analysis and understanding tasks. We present a self-supervised method for representation learning based on aligning temporal video sequences. Our framework uses a transformer-based encoder to extract frame-level features and leverages them to find the optimal alignment path between video sequences. We introduce the novel Local-Alignment Contrastive (LAC) loss, which combines a differentiable local alignment loss to capture local temporal dependencies with a contrastive loss to enhance discriminative learning. Prior works on video alignment have focused on using global temporal ordering across sequence pairs, whereas our loss encourages identifying the best-scoring subsequence alignment. LAC uses the differentiable Smith-Waterman (SW) affine method, which features a flexible parameterization learned through the training phase, enabling the model to adjust the temporal gap penalty length dynamically. Evaluations show that our learned representations outperform existing state-of-the-art approaches on action recognition tasks.

Figures (4)

• Figure 1: We introduce a representation learning approach that aligns video sequences depicting the same processes. Our training objective is to use a novel LAC loss to optimize and learn an element-wise embedding function that supports the alignment process.
• Figure 2: Our framework uses Local Alignment Loss and Contrastive Loss to optimize embeddings generated by an encoder that processes input videos after they have undergone spatio-temporal data augmentation.
• Figure 3: Similarity matrix (left) shows video alignment using an optimal path and the respective video aligned frame-by-frame (right).
• Figure 4: Fine-grained frame retrieval for Pouring (left) and PennAction (right) achieved using nearest neighbors within our embedding space.