LTOS: Layout-controllable Text-Object Synthesis via Adaptive Cross-attention Fusions

TL;DR

This paper introduces a unified task called layout-controllable text-object synthesis (LTOS), which merges text rendering and layout-based object generation, and proposes a self-adaptive cross-attention fusion that forces the image generation to attend more to important text spans.

Abstract

Controllable text-to-image generation synthesizes visual text and objects in images with certain conditions, which are frequently applied to emoji and poster generation. Visual text rendering and layout-to-image generation tasks have been popular in controllable text-to-image generation. However, each of these tasks typically focuses on single modality generation or rendering, leaving yet-to-be-bridged gaps between the approaches correspondingly designed for each of the tasks. In this paper, we combine text rendering and layout-to-image generation tasks into a single task: layout-controllable text-object synthesis (LTOS) task, aiming at synthesizing images with object and visual text based on predefined object layout and text contents. As compliant datasets are not readily available for our LTOS task, we construct a layout-aware text-object synthesis dataset, containing elaborate well-aligned labels of visual text and object information. Based on the dataset, we propose a layout-controllable text-object adaptive fusion (TOF) framework, which generates images with clear, legible visual text and plausible objects. We construct a visual-text rendering module to synthesize text and employ an object-layout control module to generate objects while integrating the two modules to harmoniously generate and integrate text content and objects in images. To better the image-text integration, we propose a self-adaptive cross-attention fusion module that helps the image generation to attend more to important text information. Within such a fusion module, we use a self-adaptive learnable factor to learn to flexibly control the influence of cross-attention outputs on image generation. Experimental results show that our method outperforms the state-of-the-art in LTOS, text rendering, and layout-to-image tasks, enabling harmonious visual text rendering and object generation.

Figures (7)

• Figure 1: Our model TOF's examples. Given predefined visual text and object information, TOF generates objects following the layout map while rendering text consistent with the image's depth, texture, and geometry.
• Figure 2: TOF framework. Given the input with a prompt, a glyph image, and a layout map, TOF achieves controllable text-object synthesis.
• Figure 3: An example sample of our LTOS dataset.
• Figure 4: Qualitative comparison of TOF with the baselines and widely used T2I generation APIs.
• Figure 5: Qualitative comparison of TOF and the baselines. Due to the differences between our input and baselines, we keep the input to be the same based on the input format accepted by all models.