Translation-based Video-to-Video Synthesis

TL;DR

This survey provides a structured taxonomy of translation-based video synthesis (TVS), distinguishing image-to-video, paired video-to-video, and unpaired video-to-video methods. It catalogs five architectural families—3D GANs, temporal-constraint, optical flow, RNNs, and extended image translators—and details their loss functions, strengths, and limitations. The paper compiles diverse datasets and extensive quantitative benchmarks, highlighting how temporal and motion representations improve realism and consistency, with neural rendering showing notable gains. It emphasizes multi-stage learning and rich evaluation protocols as key directions for achieving robust, long-range video coherence in practical TVS systems.

Abstract

Translation-based Video Synthesis (TVS) has emerged as a vital research area in computer vision, aiming to facilitate the transformation of videos between distinct domains while preserving both temporal continuity and underlying content features. This technique has found wide-ranging applications, encompassing video super-resolution, colorization, segmentation, and more, by extending the capabilities of traditional image-to-image translation to the temporal domain. One of the principal challenges faced in TVS is the inherent risk of introducing flickering artifacts and inconsistencies between frames during the synthesis process. This is particularly challenging due to the necessity of ensuring smooth and coherent transitions between video frames. Efforts to tackle this challenge have induced the creation of diverse strategies and algorithms aimed at mitigating these unwanted consequences. This comprehensive review extensively examines the latest progress in the realm of TVS. It thoroughly investigates emerging methodologies, shedding light on the fundamental concepts and mechanisms utilized for proficient video synthesis. This survey also illuminates their inherent strengths, limitations, appropriate applications, and potential avenues for future development.

Figures (9)

• Figure 1: Architecture of AffineGAN. Here, the base and residual encoders extract the content features from the input image, while the second base encoder extracts the style features from the target image. The style and content features are passed to the decoder to generate the target expression video. A discriminator plays role in determining whether the generated expression is real or fake shen2019facial.
• Figure 2: Paired v2v for video synthesis (a) Architecture of vid2vid framework. A conditional generator is used as image synthesis unit ($H$) wang2018video, (b) Architecture of the few-shot vid2vid framework. A network weight generation module $E$ is used to generate weight from example image ($e$). Modified SPADE was used for $H$wang2019few, (c) Module $E$ consists of sub-network $E_F$ for extracting features $q$ from $e$, and sub-network $E_P$ to generate wegith $\theta_H$ from $q$ for $H$wang2019few.
• Figure 3: This figure presents the architecture of vid2vid with global temporal consistency. The input video sequence is denoted as $I = [\ldots, I_{t-1}, I_t, I_{t+1}, \ldots]$, and the corresponding ground-truth is represented as $G = [\ldots, G_{t-1}, G_t, G_{t+1}, \ldots]$. The output from the generator is given by $O = [\ldots, O_{t-1}, O_t, O_{t+1}, \ldots]$. The frame $O_{t-1}$ is warped using optical flow ($W$) to produce $O'_{t}$, aligning it with $O_t$. Residual errors are computed for both the ground-truth ($E^g_t, E^g_{t+1}$) and the generated output ($E^o_t, E^o_{t+1}$). A two-channel discriminator is employed, with one channel distinguishing between the generated and ground-truth output and the other discerning the residual errors between the ground-truth and predicted videos wei2018video.
• Figure 4: 3D GAN for video synthesis. The model features two generator networks, $G_X$ and $G_Y$, designed to convert volumetric images ($X$ and $Y$) from one domain to another. Additionally, there are two discriminator networks, $D_X$ and $D_Y$, tasked with differentiating between real and synthetic videos. A key aspect of the model is its cycle consistency, ensuring that an image translated to another domain and then back again by $G_Y$($G_X$($X$)) remains identical to the original input bashkirova2018unsupervised.
• Figure 5: Illustration of the Recycle-GAN approach for video synthesis. The framework operates on two distinct but sequentially linked data streams, represented as $x = [x_1, x_2, \ldots, x_t]$ and $y = [y_1, y_2, \ldots, y_s]$. Within this setup, $G_X$ and $G_Y$ function as generators tasked with creating synthetic video frames. Complementing these, $P_X$ and $P_Y$ serve as temporal predictors, designed to recognize and utilize the temporal relationships inherent in these ordered sequences bansal2018recycle.