ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers

TL;DR

This work tackles zero-shot, general-purpose visual generation with diffusion transformers across multi-image tasks. It introduces ChatDiT, a training-free framework built on a multi-agent architecture and an in-context toolkit that operates on pretrained DiTs without fine-tuning. Evaluated on IDEA-Bench, ChatDiT achieves state-of-the-art zero-shot performance among general-purpose frameworks, while enabling features like interleaved text-image articles and multi-round conversations. Limitations remain in long-context fidelity, identity preservation, and high-level reasoning, guiding future improvements and highlighting the latent potential of diffusion models for in-context, cross-image generation.

Abstract

Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the inherent in-context generation capabilities of pretrained diffusion transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks with minimal or no architectural modifications. These capabilities are unlocked by concatenating self-attention tokens across multiple input and target images, combined with grouped and masked generation pipelines. Building upon this foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive visual generation framework that leverages pretrained diffusion transformers in their original form, requiring no additional tuning, adapters, or modifications. Users can interact with ChatDiT to create interleaved text-image articles, multi-page picture books, edit images, design IP derivatives, or develop character design settings, all through free-form natural language across one or more conversational rounds. At its core, ChatDiT employs a multi-agent system comprising three key components: an Instruction-Parsing agent that interprets user-uploaded images and instructions, a Strategy-Planning agent that devises single-step or multi-step generation actions, and an Execution agent that performs these actions using an in-context toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with diverse instructions and varying numbers of input and target images. Despite its simplicity and training-free approach, ChatDiT surpasses all competitors, including those specifically designed and trained on extensive multi-task datasets. We further identify key limitations of pretrained DiTs in zero-shot adapting to tasks. We release all code, agents, results, and intermediate outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT

Figures (5)

• Figure 1: Overview of the ChatDiT multi-agent framework. The framework consists of three core agents operating sequentially: the Instruction-Parsing Agent interprets user instructions and analyzes inputs, the Strategy-Planning Agent formulates in-context generation strategies, and the Execution Agent performs the planned actions using pretrained diffusion transformers. An optional Markdown Agent integrates the outputs into cohesive, illustrated articles. Sub-agents handle specialized tasks within each core agent, ensuring flexibility and precision in generation.
• Figure 2: Selected single-round generation examples of ChatDiT on IDEA-Bench liang2024ideabench. ChatDiT demonstrates versatility by handling diverse tasks, instructions, and input-output configurations in a zero-shot manner through free-form natural language interactions. The user messages shown here are condensed summaries of the original detailed instructions from IDEA-Bench to conserve space.
• Figure 3: Selected illustrated article generation examples of ChatDiT. ChatDiT is able to generate interleaved text-image articles based on users’ natural language instructions. It autonomously estimates the required number of images, plans and executes the generation process using in-context capabilities, and seamlessly integrates the outputs into coherent and visually engaging illustrated articles.
• Figure 4: Selected multi-round conversation examples of ChatDiT. By referencing images from the conversation history, ChatDiT is able to perform seamless multi-round generation and editing in response to free-form user instructions. This iterative process enables dynamic refinement and adaptation of outputs while maintaining contextual consistency across conversation turns. Key modifications specified in each instructional message are highlighted in yellow.
• Figure 5: Comparison of ChatDiT with existing approaches.