Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Text-Guided Convolutional Adapter for the Diffusion Model

Aryan Das, Koushik Biswas, Swalpa Kumar Roy, Badri Narayana Patro, Vinay Kumar Verma

TL;DR

The Nexus Adapters are introduced, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG), and it is demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter.

Abstract

We introduce the Nexus Adapters, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG). Recently, structure-preserving methods have achieved promising results in conditional image generation by using a base model for prompt conditioning and an adapter for structure input, such as sketches or depth maps. These approaches are highly inefficient and sometimes require equal parameters in the adapter compared to the base architecture. It is not always possible to train the model since the diffusion model is itself costly, and doubling the parameter is highly inefficient. In these approaches, the adapter is not aware of the input prompt; therefore, it is optimal only for the structural input but not for the input prompt. To overcome the above challenges, we proposed two efficient adapters, Nexus Prime and Slim, which are guided by prompts and structural inputs. Each Nexus Block incorporates cross-attention mechanisms to enable rich multimodal conditioning. Therefore, the proposed adapter has a better understanding of the input prompt while preserving the structure. We conducted extensive experiments on the proposed models and demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter. Furthermore, we also introduced a lightweight Nexus Slim adapter with 18M fewer parameters than the T2I-Adapter, which still achieved state-of-the-art results. Code: https://github.com/arya-domain/Nexus-Adapters

Efficient Text-Guided Convolutional Adapter for the Diffusion Model

TL;DR

The Nexus Adapters are introduced, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG), and it is demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter.

Abstract

We introduce the Nexus Adapters, novel text-guided efficient adapters to the diffusion-based framework for the Structure Preserving Conditional Generation (SPCG). Recently, structure-preserving methods have achieved promising results in conditional image generation by using a base model for prompt conditioning and an adapter for structure input, such as sketches or depth maps. These approaches are highly inefficient and sometimes require equal parameters in the adapter compared to the base architecture. It is not always possible to train the model since the diffusion model is itself costly, and doubling the parameter is highly inefficient. In these approaches, the adapter is not aware of the input prompt; therefore, it is optimal only for the structural input but not for the input prompt. To overcome the above challenges, we proposed two efficient adapters, Nexus Prime and Slim, which are guided by prompts and structural inputs. Each Nexus Block incorporates cross-attention mechanisms to enable rich multimodal conditioning. Therefore, the proposed adapter has a better understanding of the input prompt while preserving the structure. We conducted extensive experiments on the proposed models and demonstrated that the Nexus Prime adapter significantly enhances performance, requiring only 8M additional parameters compared to the baseline, T2I-Adapter. Furthermore, we also introduced a lightweight Nexus Slim adapter with 18M fewer parameters than the T2I-Adapter, which still achieved state-of-the-art results. Code: https://github.com/arya-domain/Nexus-Adapters
Paper Structure (19 sections, 16 equations, 4 figures, 8 tables)

This paper contains 19 sections, 16 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Background: Diffusion Models
  5. Proposed Framework Overview
  6. Nexus Adapter
  7. Nexus Blocks
  8. Prime Block
  9. Slim Block
  10. Fusion
  11. Experiment
  12. Results and Discussion
  13. Quantitative Comparison
  14. Qualitative Visualization
  15. Ablation Studies
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Architecture of our method. The framework consists of two parts: (1) A frozen Stable Diffusion model. (2) A lightweight Nexus Adapter trained to inject external condition signals. A condition image is processed through the adapter to produce multiscale features, while a text prompt is encoded by a frozen CLIP Text Encoder. Both visual and textual features are injected into the UNet via additive fusion and cross-attention. Detailed designs of Prime and Slim Nexus Blocks are shown on the right.
  • Figure 2: Qualitative comparison across four conditioning types showing that Nexus Prime consistently yields the most accurate and semantically aligned outputs, while Nexus Slim offers competitive performance with reduced complexity.
  • Figure 3: Qualitative comparison of image outputs with and without prompts, using only Sketch and Depth maps as conditional inputs.
  • Figure 4: Qualitative ablation on conflicting prompts and conditional images. The actual object in the image-conditioning input is highlighted with a blue background, and a conflicting prompt is provided as test input. We can observe that the model preserve the image-conditioned structural input while generating the object based on the text prompt.