Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation

Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, Xinchao Wang

TL;DR

This work tackles the challenge of generating multi-instance scenes with both accurate layout and high-fidelity instance features in diffusion-based text-to-image models. It introduces the Instance Feature Generation (IFG) task and the Instance Feature Adapter (IFAdapter), which uses appearance tokens and an Instance Semantic Map to align fine-grained features with exact spatial locations. A COCO-IFG benchmark and a VLM-based verification pipeline are proposed to objectively evaluate positional and feature accuracy. Experimental results show that IFAdapter improves instance feature fidelity while maintaining precise layout, and the framework is plug-and-play across various community diffusion models.

Abstract

While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances. To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models' abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.

IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation

TL;DR

This work tackles the challenge of generating multi-instance scenes with both accurate layout and high-fidelity instance features in diffusion-based text-to-image models. It introduces the Instance Feature Generation (IFG) task and the Instance Feature Adapter (IFAdapter), which uses appearance tokens and an Instance Semantic Map to align fine-grained features with exact spatial locations. A COCO-IFG benchmark and a VLM-based verification pipeline are proposed to objectively evaluate positional and feature accuracy. Experimental results show that IFAdapter improves instance feature fidelity while maintaining precise layout, and the framework is plug-and-play across various community diffusion models.

Abstract

While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances. To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models' abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.
Paper Structure (19 sections, 9 equations, 5 figures, 3 tables)

This paper contains 19 sections, 9 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Preliminaries
  5. Problem Definition
  6. IFAdapter
  7. Appearance Tokens
  8. Instance Semantic Map-Guided Generation
  9. Learning Procedure
  10. Experiments
  11. Implementation Details
  12. Experimental Setup
  13. Comparison
  14. Quantitative analysis.
  15. Qualitative analysis.
  16. ...and 4 more sections

Figures (5)

  • Figure 1: We present IFAdapter, a novel approach designed to exert fine-grained control over localized content generation in pretrained diffusion models. (a) IFAdapter has the capacity to generate intricate features with precision. (b) The plug-and-play design of IFAdapter enables it to be seamlessly applied to various community models.
  • Figure 2: Structure of proposed IFAdapter. (a) The generation process of appearance tokens. For simplicity, we use the generation process of one instance (the corgi) as example. (b) The construction process of the Instance Semantic Map.
  • Figure 3: Qualitative results. We compare the models’ ability to generate instances with different types of features, including mixed colors, varied materials, and intricate textures.
  • Figure 4: The IFAdapter can seamlessly integrate with community diffusion models.
  • Figure 5: Qualitative results of variants of IFAdapter.