Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning

Qiang Zhu, Kuan Lu, Menghao Huo, Yuxiao Li

TL;DR

This paper addresses paired image-to-image translation by replacing traditional GAN-based backbones with a CLIP-conditioned Diffusion Transformer operating in a latent space. By conditioning on image embeddings from a pre-trained CLIP encoder and guiding denoising with a fusion of reconstruction, perceptual, and semantic losses, the method achieves semantically faithful and stylistically coherent translations on real2comic and edges2shoes tasks. The approach demonstrates superior translation quality and structural preservation over GAN baselines, at the cost of higher computational requirements due to diffusion sampling and transformer-scale modeling. The work provides a GAN-free, scalable alternative for high-fidelity I2I translation and outlines future work on unpaired translation and further efficiency improvements.

Abstract

Image-to-image translation aims to learn a mapping between a source and a target domain, enabling tasks such as style transfer, appearance transformation, and domain adaptation. In this work, we explore a diffusion-based framework for image-to-image translation by adapting Diffusion Transformers (DiT), which combine the denoising capabilities of diffusion models with the global modeling power of transformers. To guide the translation process, we condition the model on image embeddings extracted from a pre-trained CLIP encoder, allowing for fine-grained and structurally consistent translations without relying on text or class labels. We incorporate both a CLIP similarity loss to enforce semantic consistency and an LPIPS perceptual loss to enhance visual fidelity during training. We validate our approach on two benchmark datasets: face2comics, which translates real human faces to comic-style illustrations, and edges2shoes, which translates edge maps to realistic shoe images. Experimental results demonstrate that DiT, combined with CLIP-based conditioning and perceptual similarity objectives, achieves high-quality, semantically faithful translations, offering a promising alternative to GAN-based models for paired image-to-image translation tasks.

Image-to-Image Translation with Diffusion Transformers and CLIP-Based Image Conditioning

TL;DR

This paper addresses paired image-to-image translation by replacing traditional GAN-based backbones with a CLIP-conditioned Diffusion Transformer operating in a latent space. By conditioning on image embeddings from a pre-trained CLIP encoder and guiding denoising with a fusion of reconstruction, perceptual, and semantic losses, the method achieves semantically faithful and stylistically coherent translations on real2comic and edges2shoes tasks. The approach demonstrates superior translation quality and structural preservation over GAN baselines, at the cost of higher computational requirements due to diffusion sampling and transformer-scale modeling. The work provides a GAN-free, scalable alternative for high-fidelity I2I translation and outlines future work on unpaired translation and further efficiency improvements.

Abstract

Image-to-image translation aims to learn a mapping between a source and a target domain, enabling tasks such as style transfer, appearance transformation, and domain adaptation. In this work, we explore a diffusion-based framework for image-to-image translation by adapting Diffusion Transformers (DiT), which combine the denoising capabilities of diffusion models with the global modeling power of transformers. To guide the translation process, we condition the model on image embeddings extracted from a pre-trained CLIP encoder, allowing for fine-grained and structurally consistent translations without relying on text or class labels. We incorporate both a CLIP similarity loss to enforce semantic consistency and an LPIPS perceptual loss to enhance visual fidelity during training. We validate our approach on two benchmark datasets: face2comics, which translates real human faces to comic-style illustrations, and edges2shoes, which translates edge maps to realistic shoe images. Experimental results demonstrate that DiT, combined with CLIP-based conditioning and perceptual similarity objectives, achieves high-quality, semantically faithful translations, offering a promising alternative to GAN-based models for paired image-to-image translation tasks.

Paper Structure

This paper contains 20 sections, 9 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Denoising Diffusion Probabilistic Models (DDPMs)
  4. Latent Diffusion Models (LDMs)
  5. Transformers and Vision Transformers (ViT)
  6. Diffusion Transformer (DiT)
  7. Method
  8. Data Pre-processing
  9. Architecture
  10. Loss Function Design
  11. Reconstruction Loss.
  12. Perceptual Loss (LPIPS).
  13. Semantic Consistency Loss (CLIP).
  14. Experiments
  15. Dataset
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Image-Conditioned Diffusion Transformer Architecture
  • Figure 2: The training loss over iteration steps
  • Figure 3: Qualitative examples from the unseen edges2shoes dataset. Our method produces sharper, more detailed, and more realistic images compared to Pix2Pix and Pix2PixHD.
  • Figure 4: Qualitative examples from the unseen face2comics dataset.The CLIP-conditioned DiT better preserves facial identity and stylization details compared to Pix2Pix and Pix2PixHD.