Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Qihoo-T2X: An Efficient Proxy-Tokenized Diffusion Transformer for Text-to-Any-Task

Jing Wang, Ao Ma, Jiasong Feng, Dawei Leng, Yuhui Yin, Xiaodan Liang

TL;DR

The Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention to model global visual information efficiently, and further develops the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks.

Abstract

The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to model global visual information efficiently. Specifically, within each transformer block, we compute an averaging token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing the computational complexity in both image and video generation tasks (e.g., a 49% reduction compared to DiT and a 34% reduction compared to PixArt-$α$). The visual exhibition and source code of Qihoo-T2X is available at https://360cvgroup.github.io/Qihoo-T2X/.

Qihoo-T2X: An Efficient Proxy-Tokenized Diffusion Transformer for Text-to-Any-Task

TL;DR

The Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention to model global visual information efficiently, and further develops the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks.

Abstract

The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to model global visual information efficiently. Specifically, within each transformer block, we compute an averaging token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing the computational complexity in both image and video generation tasks (e.g., a 49% reduction compared to DiT and a 34% reduction compared to PixArt-). The visual exhibition and source code of Qihoo-T2X is available at https://360cvgroup.github.io/Qihoo-T2X/.
Paper Structure (22 sections, 4 equations, 10 figures, 4 tables)

This paper contains 22 sections, 4 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Image Generation with Diffusion Transformer.
  4. Video Generation with Diffusion Transformer.
  5. Efficient Diffusion Transformer.
  6. Method
  7. Redundancy Analysis
  8. Architecture of PT-DiT
  9. Global Information Interaction Module
  10. Texture Complement Module
  11. Compression Ratios
  12. Complexity Analysis
  13. Experiment
  14. Experimental Setup
  15. Qualitative analysis
  16. ...and 7 more sections

Figures (10)

  • Figure 1: The samples from Qihoo-T2I showcase high fidelity and aesthetic qualities, demonstrating a strong consistency with given textual descriptions.
  • Figure 2: Comparison of complexity between PixArt-$\alpha$ and PT-DiT/L at various resolutions.
  • Figure 3: The attention map of self-attention in PixArt-$\alpha$ at 512 resolution. We assemble the attention map for 16 tokens within a $4 \times 4$ spatial window. The vertical axis represents different tokens within the window, and the horizontal axis represents their correlation with all latent tokens. It is evident that the attention of different tokens in the same window is almost identical for spatially distant tokens, whereas there is noticeable variation for spatially neighboring tokens.
  • Figure 4: The overall architecture of PT-DiT. The image or video undergoes processing through a 3D VAE, followed by noise addition, patch embedding, and positional encoding to generate latent tokens. We replace global attention with proxy-tokenized attention to establish contextual associations and employ visual cross-attention to propagate this information to all tokens, thereby reducing computational redundancy. Moreover, texture detail modeling is enhanced through window attention and shifted window attention.
  • Figure 5: Qualitative comparison of Text-to-Image generation models.
  • ...and 5 more figures