Paying U-Attention to Textures: Multi-Stage Hourglass Vision Transformer for Universal Texture Synthesis

TL;DR

This work introduces U-Attention, a vision Transformer framework with a hierarchical hourglass backbone for universal texture synthesis that scales patches across multiple levels of detail. By combining an encoder, multi-stage Transformer blocks with down/up convolutions, and skip connections, it enables coarse-to-fine-to-coarse patch mapping and efficient single-pass synthesis that generalizes to unseen textures. The model employs a composite loss including $\mathcal{L}_{l_1}$, perceptual, style, and temporal-patch GAN terms, and achieves practical inference times on large texture datasets. Overall, the approach demonstrates $2\times$ texture synthesis across diverse textures with strong structural and perceptual quality, and advances attention-based image-to-image mapping for texture generation.

Abstract

We present a novel U-Attention vision Transformer for universal texture synthesis. We exploit the natural long-range dependencies enabled by the attention mechanism to allow our approach to synthesize diverse textures while preserving their structures in a single inference. We propose a hierarchical hourglass backbone that attends to the global structure and performs patch mapping at varying scales in a coarse-to-fine-to-coarse stream. Completed by skip connection and convolution designs that propagate and fuse information at different scales, our hierarchical U-Attention architecture unifies attention to features from macro structures to micro details, and progressively refines synthesis results at successive stages. Our method achieves stronger 2$\times$ synthesis than previous work on both stochastic and structured textures while generalizing to unseen textures without fine-tuning. Ablation studies demonstrate the effectiveness of each component of our architecture.

Figures (14)

• Figure 1: Proposed U-Attention framework with hierarchical hourglass Transformers. We introduce a multi-scale partition of the feature map between hierarchical Transformer blocks to form input patches of different scales for different Transformers. The input texture image is first projected into feature space by an encoder. We then leverage a succession of Transformer blocks, with up and down convolutions in between (purple arrows), processing the feature maps at different resolutions. Each Transformer block takes the whole feature maps as input, and we partition the feature maps to be sequences of patches of progressively smaller or larger sizes at consecutive stages of the network. Therefore, the input patch size of all the stages forms an hourglass-like scale change (dotted blue line), enabling attention to finer/coarser details at different attention steps. Finally, we add skip connections that propagate and concatenate outputs from different previous stages as part of the inputs for later Transformer stages (yellow arrows).
• Figure 1: Comparison to previous works demonstrates that our approach can handle textures of varying patterns and structures, and preserve the coherent color shift and structural details.
• Figure 2: Details of a single Transformer block (T-block). An input latent feature map is partitioned into a sequence of patches, and the input patch sequence is mapped to another patch sequence via a stack of 2 Transformer layers. The output sequence is then reshaped back into a whole feature map.
• Figure 2: Synthesis results of our U-Attention network demonstrate that our approach generalizes to a broad range of textures with varieties of randomness and structure using one trained network.
• Figure 3: Details of a transition between two Transformer stages that formulate the change of input patch scales for different Transformers. A feature map formed after a Transformer block (T-Block1) is downscaled by 2$\times$ with 4$\times$ enlarged channel dimension using a strided convolution, and then partitioned into 4$\times$ more patches as the input for the next Transformer block. Because the spatial extent is halved by the strided convolution, and the number of patches is doubled along each spatial dimension with the new partition, the input patches for the next Transformer block (T-Block2) are 4$\times$ smaller in each spatial dimension compared to the input patches for T-Block1. Each patch effectively represents 2$\times$ smaller regions than the previous step.