Text-Guided Synthesis of Eulerian Cinemagraphs

TL;DR

This work tackles the challenge of generating cinemagraphs from text prompts, including imaginary and artistic styles, by introducing a twin-image synthesis framework that pairs an artistic image with a structurally aligned realistic twin. A mask-guided flow predictor leverages segmentation and diffusion-attention features to generate plausible motion, which is then transferred to the artistic image to produce looping cinemagraphs. The approach is validated on real-world and artistic domains, showing improved motion realism and temporal coherence over strong baselines, with user studies favoring the results. Extensions demonstrate animating paintings and text-driven control of motion direction, highlighting practical utility for content creation with reduced manual effort.

Abstract

We introduce Text2Cinemagraph, a fully automated method for creating cinemagraphs from text descriptions - an especially challenging task when prompts feature imaginary elements and artistic styles, given the complexity of interpreting the semantics and motions of these images. We focus on cinemagraphs of fluid elements, such as flowing rivers, and drifting clouds, which exhibit continuous motion and repetitive textures. Existing single-image animation methods fall short on artistic inputs, and recent text-based video methods frequently introduce temporal inconsistencies, struggling to keep certain regions static. To address these challenges, we propose an idea of synthesizing image twins from a single text prompt - a pair of an artistic image and its pixel-aligned corresponding natural-looking twin. While the artistic image depicts the style and appearance detailed in our text prompt, the realistic counterpart greatly simplifies layout and motion analysis. Leveraging existing natural image and video datasets, we can accurately segment the realistic image and predict plausible motion given the semantic information. The predicted motion can then be transferred to the artistic image to create the final cinemagraph. Our method outperforms existing approaches in creating cinemagraphs for natural landscapes as well as artistic and other-worldly scenes, as validated by automated metrics and user studies. Finally, we demonstrate two extensions: animating existing paintings and controlling motion directions using text.

Figures (12)

• Figure 1: Our method can synthesize artistic cinemagraphs given text prompts, bringing to life motion effects such as "waterfall falling", "river flowing", and "turbulent ocean." These visual effects may be challenging to depict in a static photograph, but they flourish in the medium of cinemagraphs. To view the teaser images as videos, we recommend using Adobe Acrobat.
• Figure 2: Overview. Given a text prompt ${\boldsymbol{c}}\xspace$, we generate twin images with Stable Diffusion, an artistic image ${\boldsymbol{x}}\xspace$ in the style described in the text prompt, and a realistic counterpart $\hat{{\boldsymbol{x}}}\xspace$ using the modified prompt $\hat{{\boldsymbol{c}}}\xspace$. Twin images share a similar semantic layout. We then extract a binary mask ${\boldsymbol{M}}\xspace$ of the moving regions from the Self-Attention maps obtained during the artistic image's generation process. We use the mask and the realistic image to predict the optical flow $\hat{{\boldsymbol{F}}}\xspace$ with the flow prediction model ${\boldsymbol{G}}_{flow}\xspace$. Since the twin images have a very similar semantic layout, we can use the flow $\hat{{\boldsymbol{F}}}\xspace$ to animate the artistic image, with our video generator ${\boldsymbol{G}}_{frame}\xspace$.
• Figure 3: ODISE vs. Self-Attention Mask. We aim to derive a mask, given twin images: (a) the generated artistic image and (b) its realistic counterpart. Directly applying a segmentation model (e.g., ODISE xu2023open) to the realistic image might introduce segmentation errors. For example, in the ODISE-generated mask on the artistic image (c), some regions in the mask overlap with hills, which can result in the movement of the hills. We use the diffusion model's self-attention maps to further refine the mask (d).
• Figure 4: Mask generation pipeline. To generate our final binary mask ${\boldsymbol{M}}\xspace$, we first generate a realistic twin (b) corresponding to the artistic image (a). Using a pre-trained open set panoptic segmentation module, ODISE xu2023open, and a user-specified region name 'river', we generate a binary segmentation mask (c). In (d), we use PCA pca to visualize the self-attention maps averaged across different timesteps. Finally, we apply spectral clustering ng2001spectral on the average self-attention maps (e) and select regions with high Intersection-over-Union (IOU) scores with respect to the ODISE mask (c).
• Figure 5: Visual comparisons for optical flow prediction. We compare our method with two single-image animation methods Holynski et al. holynski2021animating and SLR-SFS fan2022simulating on real videos (a) and artistic images (b). For real videos, we also show the ground truth optical flow averaged across all frames. Overall, our method predicts more plausible motions that align better with target regions. Please see our supplement for more video comparisons with more baselines.