Semantix: An Energy Guided Sampler for Semantic Style Transfer

TL;DR

Semantix presents a training-free energy-guided sampler for semantic style transfer that leverages semantic correspondence to guide style and appearance transfer across images and videos supported by pre-trained diffusion models. It inverts content to the diffusion noise space using DDPM inversion and applies a three-term energy function—Style Feature Guidance, Spatial Feature Guidance, and Semantic Distance—to steer sampling without modifying the base diffusion model. The approach is compatible with both image and video pipelines, preserving structure and motion continuity while achieving semantically aligned feature transfer. Experimental results show improvements over state-of-the-art baselines in structure preservation, style fidelity, and visual quality, with user studies corroborating perceptual gains. Limitations include potential failures when the context has strong intrinsic style and reliance on diffusion-model semantics, suggesting future work toward newer architectures like UViT or DiT.

Abstract

Recent advances in style and appearance transfer are impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspondence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, Semantic Style Transfer, which involves transferring style and appearance features from a reference image to a target visual content based on semantic correspondence. We subsequently propose a training-free method, Semantix an energy-guided sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, as a sampler, Semantix be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, Semantix utilizes a meticulously crafted energy function to guide the sampling process, including three key components: Style Feature Guidance, Spatial Feature Guidance and Semantic Distance as a regularisation term. Experimental results demonstrate that Semantix not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields. The project website is available at https://huiang-he.github.io/semantix/

Figures (19)

• Figure 1: Examples of our Semantix. Given a visual context and a reference image (Top examples), Semantix can perform Semantic Style Transfer based on the semantic correspondence. Besides, our Semantix also can be directly adapted for the videos (Bottom examples) without the need of additional modification. It is important to emphasize that, as a sampler, Semantix directly leverages the knowledge from the pretrained model to guide the sampling process based on our proposed energy function for Semantic Style Transfer, without the need for any additional training or optimization.
• Figure 2: Overview of Semantix. Given a reference image $I^{ref}$ and a context image $I^{c}$ or video $V^c$, we first invert them to the latent $x_T$ through an edit-friendly DDPM inversion. In the denoising process, we then modify the $x_t^{out}$ through the designed energy gradient in every sampling step.
• Figure 3: Visualizing feature maps. We extracted features from the second block of the diffusion model decoder and visualized the top three PCA components and feature mapping at each timestep.
• Figure 4: Qualitative comparison with style transfer and appearance transfer methods. The top two rows are comparisons of style transfer, the bottom two of appearance transfer.
• Figure 5: Qualitative comparison of video style transfer. Click the images to play the animation clips. (Recommended to use Adobe Reader to ensure the GIFs play properly.)