PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models

TL;DR

PIA, a Personalized Image Animator that excels in aligning with condition images, achieving motion controllability by text, and the compatibility with various personalized T2I models without specific tuning is presented.

Abstract

Recent advancements in personalized text-to-image (T2I) models have revolutionized content creation, empowering non-experts to generate stunning images with unique styles. While promising, adding realistic motions into these personalized images by text poses significant challenges in preserving distinct styles, high-fidelity details, and achieving motion controllability by text. In this paper, we present PIA, a Personalized Image Animator that excels in aligning with condition images, achieving motion controllability by text, and the compatibility with various personalized T2I models without specific tuning. To achieve these goals, PIA builds upon a base T2I model with well-trained temporal alignment layers, allowing for the seamless transformation of any personalized T2I model into an image animation model. A key component of PIA is the introduction of the condition module, which utilizes the condition frame and inter-frame affinity as input to transfer appearance information guided by the affinity hint for individual frame synthesis in the latent space. This design mitigates the challenges of appearance-related image alignment within and allows for a stronger focus on aligning with motion-related guidance.

Figures (17)

• Figure 1: Given an elaborated image generated by a personalized text-to-image model, the proposed Personalized Image Animator (PIA) animates it with realistic motions according to different text prompts while preserving the original distinct styles and high-fidelity details. We recommend using Abode Arobat and clicking the images to play the animation clips. [Best viewed in color with zoom-in]
• Figure 2: Personalized Image Animator (PIA). As shown in (a), PIA consists of a text-to-image (T2I) model, well-trained temporal alignment layers (TA), and a new condition module $\mathcal{W}_{cond}$ responsible for encoding the condition image $z^I$ and inter-frame affinity $s^{1:F}$. In particular, the T2I model consists of U-Net blocks, including a ResBlock (Res) he2016resnet, a self-attention layer (SA), and a cross-attention layer (CA), as depicted in (b). During training, the condition module learns to leverage the affinity hints and incorporate appearance information from the condition images, facilitating image alignment and enabling a stronger emphasis on motion-related alignment.
• Figure 3: Illustration of the condition module (CM). A vanilla personalized T2V model (shown in (a)) needs to align both the appearance and motion of individual frames simultaneously. PIA with CM (shown in (b)) can borrow appearance information from condition image $z^I$ with affinity hints $s$, easing the challenge of both appearance and motion alignment. We use the color and strip to denote appearance and motion, respectively.
• Figure 4: An example of AnimateBench. The images in AnimateBench are carefully crafted using a set of collected personalized text-to-image models. Each image has three carefully designed prompts, describing the following motions that the image probably happens within a single short shot.
• Figure 5: Qualitative comparison with state-of-the art approaches. Compared with other methods, PIA shows excellent motion controllability and strong image alignment. Specifically, in the first case, PIA generates a "walking" motion for the toy bear (in its feet), while other methods can only remain static frames, showing a lack of motion controllability. In the second case, PIA adds a new element, i.e., fire, with realistic motion. We show more video cases in supplementary materials due to the file size limit of the main paper.