Synchronization of Multiple Videos

TL;DR

The paper tackles the problem of synchronizing multiple videos of the same action captured in different scenes or generated by AI, where simple time shifts are insufficient. It introduces Temporal Prototype Learning (TPL), which learns a shared 1D temporal prototype by encoding frame sequences into univariate latent signals $Z_i$ and aligning them to a common prototype $\widehat{Z}$, then reconstructs back to the original embeddings. The core contributions are a prototype-based joint synchronization framework, the Diffeomorphic Multitasking Autoencoder (D-MTAE), and a linear-time frame retrieval mechanism, plus demonstrating synchronization of GenAI videos on the GenAI-MVS dataset. Experiments show TPL outperforms traditional multi-sequence alignment baselines across Penn Action, Pouring, Internet Videos, and GenAI-MVS, and generalizes to AI-generated content, enabling robust annotation transfer for large-scale video analytics.

Abstract

Synchronizing videos captured simultaneously from multiple cameras in the same scene is often easy and typically requires only simple time shifts. However, synchronizing videos from different scenes or, more recently, generative AI videos, poses a far more complex challenge due to diverse subjects, backgrounds, and nonlinear temporal misalignment. We propose Temporal Prototype Learning (TPL), a prototype-based framework that constructs a shared, compact 1D representation from high-dimensional embeddings extracted by any of various pretrained models. TPL robustly aligns videos by learning a unified prototype sequence that anchors key action phases, thereby avoiding exhaustive pairwise matching. Our experiments show that TPL improves synchronization accuracy, efficiency, and robustness across diverse datasets, including fine-grained frame retrieval and phase classification tasks. Importantly, TPL is the first approach to mitigate synchronization issues in multiple generative AI videos depicting the same action. Our code and a new multiple video synchronization dataset are available at https://bgu-cs-vil.github.io/TPL/

Figures (9)

• Figure 1: Temporal Prototype Learning (TPL) uses an 'off-the-shelf' feature extractor, denoted by $\phi$, to generate initial multichannel action progression sequences for videos of the same action (e.g., Ball pitch zhang:ICCV:2013:penn). Colors indicate different (and temporally-misaligned) videos of the same action. TPL produces the joint alignment and prototypical sequence, mapping key events (e.g., Ball Release)
• Figure 2: Diffeomorphic Multitasking Autoencoder (D-MTAE) for Temporal Prototype Learning, consists of: 1) $\Psi_{\text{enc}}$, an encoder for dimensionality reduction; 2) $\Psi_{\text{Align}}$Shapira:NIPS:2019:DTAN, for joint alignment; and 3) $\Psi_{\text{Dec}}$, a decoder. The losses for JA and DR are $\mathcal{L}_{ICAE}$ and $\mathcal{L}_{rec}$ respectively. The feature extractor, $\phi$, could either by trained per dataset (e.g., CARL Chen:CVPR:2022:CARL) or a pretrained foundation model (e.g., DINO Caron:ICCV:2021:dino).
• Figure 3: Univariate representations learned by TPL for 20 videos depicting a Baseball swing colored by the phase labels, before (top) and after Synchronization (bottom).
• Figure 4: Examples from our GenerativeAI Multiple Video Synchronization (GenAI-MVS) dataset, showing seven equally spaced frames before (top) and after (bottom) synchronization. The first video (left) depicts a "monkey doing dips," and the second video (right) shows a "bear performing a deadlift." We highlight mismatches in the original videos in red, and TPL matching in green. In both cases, alignment via TPL successfully synchronizes the key phases of the action progression.
• Figure 5: Joint alignment of the one-dimensional representation for 20 randomly sampled videos of the Baseball swing (top) and golf swing (bottom) action colored by the phase labels, before (left) and after alignment (right).