Learning Graph Representation of Agent Diffusers

TL;DR

This work addresses the variability of diffusion-model performance across datasets by introducing LGR-AD, a graph-based, multi-agent diffusion framework. It models the generation process as interacting agent diffusers connected via a model graph, learns optimal coordination with a Graph Convolutional Network operating on a maximum spanning tree of agent relationships, and optimizes a hybrid loss that balances accuracy, diversity, and graph structure through a Laplacian regularization term. The approach yields theoretical insights and empirical gains, with the GCNN meta-model leveraging MST-guided connectivity to produce coherent, high-fidelity images across diverse benchmarks; code is provided. Overall, LGR-AD demonstrates that structured, graph-informed coordination of expert diffusers can deliver scalable, flexible, and robust text-to-image synthesis, paving the way for adaptive image generation in complex tasks.

Abstract

Diffusion-based generative models have significantly advanced text-to-image synthesis, demonstrating impressive text comprehension and zero-shot generalization. These models refine images from random noise based on textual prompts, with initial reliance on text input shifting towards enhanced visual fidelity over time. This transition suggests that static model parameters might not optimally address the distinct phases of generation. We introduce LGR-AD (Learning Graph Representation of Agent Diffusers), a novel multi-agent system designed to improve adaptability in dynamic computer vision tasks. LGR-AD models the generation process as a distributed system of interacting agents, each representing an expert sub-model. These agents dynamically adapt to varying conditions and collaborate through a graph neural network that encodes their relationships and performance metrics. Our approach employs a coordination mechanism based on top-$k$ maximum spanning trees, optimizing the generation process. Each agent's decision-making is guided by a meta-model that minimizes a novel loss function, balancing accuracy and diversity. Theoretical analysis and extensive empirical evaluations show that LGR-AD outperforms traditional diffusion models across various benchmarks, highlighting its potential for scalable and flexible solutions in complex image generation tasks. Code is available at: https://github.com/YousIA/LGR_AD

Figures (2)

• Figure 1: LGR-AD begins by training diffusion-based models for text-to-image generation, treating each model as an agent in a multi-agent system. After training each agent diffuser , we use the agents' outputs and specifications to construct a graph, where nodes represent agents and edges capture their interactions. GCNN is then applied to learn an optimal representation of the graph, leveraging agent collaboration. The learned features are processed through a fully connected layer to guide image generation, enabling the system to adapt and refine the diffusion process for improved fidelity and coherence.
• Figure 2: Selected qualitative results of LGR-AD compared to the baseline solutions.

Theorems & Definitions (7)

• definition 1: Model Output
• definition 2: Model Specification
• definition 3: Connectivity Function
• definition 4: Characteristic Connectivity Function
• definition 5: Performance Connectivity Function
• definition 6: Graph of Models
• definition 7: Loss Function