FlowTok: Flowing Seamlessly Across Text and Image Tokens

TL;DR

FlowTok tackles cross-modal generation by unifying text and image into a compact 1D latent space and learning a direct flow between modalities using flow matching, avoiding diffusion-based conditioning and 2D latents. It introduces a 1D image tokenizer augmented with RoPE and SwiGLU, a text projector to align CLIP embeddings, and KL regularization plus a text alignment loss to preserve semantics. The framework achieves competitive text-to-image and image-to-text results while drastically reducing training costs and enabling faster sampling compared to state-of-the-art methods. Demonstrating strong results on public datasets, FlowTok shows that 1D token flows can provide practical efficiency gains for cross-modal generation and generalize to image-to-text tasks.

Abstract

Bridging different modalities lies at the heart of cross-modality generation. While conventional approaches treat the text modality as a conditioning signal that gradually guides the denoising process from Gaussian noise to the target image modality, we explore a much simpler paradigm-directly evolving between text and image modalities through flow matching. This requires projecting both modalities into a shared latent space, which poses a significant challenge due to their inherently different representations: text is highly semantic and encoded as 1D tokens, whereas images are spatially redundant and represented as 2D latent embeddings. To address this, we introduce FlowTok, a minimal framework that seamlessly flows across text and images by encoding images into a compact 1D token representation. Compared to prior methods, this design reduces the latent space size by 3.3x at an image resolution of 256, eliminating the need for complex conditioning mechanisms or noise scheduling. Moreover, FlowTok naturally extends to image-to-text generation under the same formulation. With its streamlined architecture centered around compact 1D tokens, FlowTok is highly memory-efficient, requires significantly fewer training resources, and achieves much faster sampling speeds-all while delivering performance comparable to state-of-the-art models. Code is available at https://github.com/TACJu/FlowTok.

Figures (4)

• Figure 1: Text-to-Image Generation Results by FlowTok. FlowTok projects both text and images into a unified, compact 1D latent space, enabling direct flow matching between 1D tokens and facilitating the efficient generation of diverse, high-fidelity images.
• Figure 2: Text as Conditions vs. Direct Flow between Modalities.Top: Conventional text-to-image generation relies on the diffusion process, where text serves as a conditioning signal to guide the denoising process. Bottom: The proposed FlowTok enables direct flow between text and image modalities by projecting both into a shared, compact 1D latent space, facilitating seamless generation of both.
• Figure 3: COCO Results. FlowTok presents comparable performance to previous methods on COCO while significantly reducing training resource requirements (Fig. \ref{['fig:fidvst']}) and achieving much faster sampling speed (Fig. \ref{['fig:fidvsi']}). This efficiency stems from its minimalist design centered around 1D tokens, which facilitates direct transformation between text and image modalities, leading to superior performance with enhanced computational efficiency. We note that the compared CrossFlow liu2024flowing uses high-quality proprietary data.
• Figure 4: Overview of FlowTok. FlowTok is a minimal framework that facilitates seamless flow between 1D text tokens and image tokens for both text-to-image and image-to-text generation. Top: For text-to-image generation, the input text is encoded by the CLIP text encoder into $\mathbf{T}_{\text{init}} \in \mathbb{R}^{N \times C}$, projected into a low-dimensional latent space as text tokens $\mathbf{Z}_{\text{T}} \in \mathbb{R}^{N \times D}$, then transformed into image tokens $\mathbf{Z}_{\text{I}} \in \mathbb{R}^{N \times D}$ of the same shape through flow matching and decoded by a 1D Image VAE Decoder to generate the final image. Bottom: For image-to-text generation, an input image is encoded by a 1D Image VAE Encoder into $\mathbf{Z}_{\text{I}}$, mapped to $\mathbf{Z}_{\text{T}}$ through flow matching and decoded into text via a text decoder. Unlike conventional approaches that rely on 2D noise and image latents (e.g., $32 \times 32 \times 4$ for 256-resolution images) with text as conditions, our direct 1D transformation (i.e., $77 \times 16$) achieves a 3.3$\times$ compression rate, significantly reducing memory costs, accelerating training, and enabling faster inference.