ITTR: Unpaired Image-to-Image Translation with Transformers

TL;DR

ITTR addresses unpaired image-to-image translation by introducing a transformer-based generator with two key innovations: Hybrid Perception Block (HPB), which fuses local and global context, and Dual Pruned Self-Attention (DPSA), which reduces attention computation through row/column token pruning. The approach enables efficient long-range dependency modeling without the prohibitive costs of standard MHSA, achieving state-of-the-art results on six benchmarks with improved FID and DRN-Score while maintaining content fidelity. The work demonstrates that generative vision transformers can outperform CNN-based generators in unpaired translation when equipped with multi-scale contextual fusion and token-sparse attention. Practically, ITTR offers a scalable, high-quality alternative for cross-domain image translation with potential impact on tasks requiring global coherence and fine-grained detail preservation.

Abstract

Unpaired image-to-image translation is to translate an image from a source domain to a target domain without paired training data. By utilizing CNN in extracting local semantics, various techniques have been developed to improve the translation performance. However, CNN-based generators lack the ability to capture long-range dependency to well exploit global semantics. Recently, Vision Transformers have been widely investigated for recognition tasks. Though appealing, it is inappropriate to simply transfer a recognition-based vision transformer to image-to-image translation due to the generation difficulty and the computation limitation. In this paper, we propose an effective and efficient architecture for unpaired Image-to-Image Translation with Transformers (ITTR). It has two main designs: 1) hybrid perception block (HPB) for token mixing from different receptive fields to utilize global semantics; 2) dual pruned self-attention (DPSA) to sharply reduce the computational complexity. Our ITTR outperforms the state-of-the-arts for unpaired image-to-image translation on six benchmark datasets.

Figures (7)

• Figure 1: Comparison between ITTR and CUT cut with the Grad-CAM gcam visualization extracted from the first three blocks in the body of the generator. The bottom colorbar displays the relative saliency weight value of Grad-CAM, of which a red area indicates more contribution to the image-to-image translation and a blue one the opposite. By comparison, the transformer-based ITTR is more likely to capture instance-level contextual information than the CNN-based CUT.
• Figure 2: Qualitative comparison of I2ISA i2i_sa and ITTR. Though both of the methods have integrated self-attention into their network architecture, translation performance of the former is far behind the latter.
• Figure 3: Examples of translation results produced by ITTR. From left to right, top to bottom: Horse $\rightarrow$ Zebra, Cat $\rightarrow$ Dog, Selfie $\rightarrow$ Anime, Face $\rightarrow$ MetFace, Female $\rightarrow$ Cartoon, Cityscapes.
• Figure 4: The architecture of ITTR. From left to right: the overall architecture of ITTR, the architecture of hybrid perception block (HPB), dual pruned self-attention (DPSA). "Conv" means convolution. "IN" is the abbreviation of instance normalization. "DW" means depth-wise. "C" in the circle represents concatenation. "L2 Norm" is token-wise L2 normalization. The spatial-wise sum is to calculate the summation of all the tokens in $Q$. Row-wise or Col-wise sum is to calculate the summation of tokens in the same row or column in $K$. Token selection is to select rows and columns from a matrix with indexes for referring. "$\times$" in the circle represents matrix multiplication. The dotted box in right represents operations made in a single head of DPSA.
• Figure 5: Schematic diagram of token pre-pruning mechanism in DPSA as described in Section \ref{['sec:dpsa']}. Tokens in Key are grouped by rows and columns to compute contribution score. Tokens in rows and columns with less contribution are pruned from Key and Value.