Inference-based GAN Video Generation

TL;DR

The paper tackles the challenge of generating long, temporally coherent videos with high visual fidelity. It introduces EncGAN3, a hybrid VAE-GAN framework with dual content and movement streams and an encoder-driven inference pathway, enabling controllable generation of short clips. To extend to long videos, it presents a memory-efficient recall mechanism (REncGAN3) that stitches short clips via a Markov-chain-based recall using a reference frame, achieving hundreds to thousands of frames without proportional memory growth. Experimental results across multiple action and scene datasets demonstrate superior FID/IS on short videos and favorable FVD/recall-based metrics for long videos, with ablations confirming the importance of encoder, dual streams, and the recall framework.

Abstract

Video generation has seen remarkable progresses thanks to advancements in generative deep learning. Generated videos should not only display coherent and continuous movement but also meaningful movement in successions of scenes. Generating models such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) and more recently Diffusion Networks have been used for generating short video sequences, usually of up to 16 frames. In this paper, we first propose a new type of video generator by enabling adversarial-based unconditional video generators with a variational encoder, akin to a VAE-GAN hybrid structure, in order to enable the generation process with inference capabilities. The proposed model, as in other video deep learning-based processing frameworks, incorporates two processing branches, one for content and another for movement. However, existing models struggle with the temporal scaling of the generated videos. In classical approaches when aiming to increase the generated video length, the resulting video quality degrades, particularly when considering generating significantly long sequences. To overcome this limitation, our research study extends the initially proposed VAE-GAN video generation model by employing a novel, memory-efficient approach to generate long videos composed of hundreds or thousands of frames ensuring their temporal continuity, consistency and dynamics. Our approach leverages a Markov chain framework with a recall mechanism, with each state representing a VAE-GAN short-length video generator. This setup allows for the sequential connection of generated video sub-sequences, enabling temporal dependencies, resulting in meaningful long video sequences.

Figures (16)

• Figure 1: The architecture of EncGAN3: two Encoders ($Enc$), three-stream Generator ($G$) and two Discriminators ($D$) for content and movement.
• Figure 2: Illustration for the Recall EncGAN3 (REncGAN3) long-term video generation method.
• Figure 3: Generated video frames of $64 \times 64$ pixels resolution by EncGAN3.
• Figure 4: Comparing EncGAN3 ( upper panels) with G$^3$AN ( bottom panels) after being trained on various datasets where the generated videos have a resolution of $64 \times 64$ pixels.
• Figure 5: The convergence during the training, indicated by the FID scores evaluated for every 100 epochs, for EncGAN3 ( solid lines) and G$^3$AN ( dashed lines) when training on Weizmann and UCF101 datasets.