Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation

TL;DR

This work introduces a benchmark, LenCom-Eval, specifically designed for testing models' capability in generating images with Lengthy and Complex visual text, and introduces a training-free framework to enhance the two-stage generation approaches.

Abstract

Over the past few years, Text-to-Image (T2I) generation approaches based on diffusion models have gained significant attention. However, vanilla diffusion models often suffer from spelling inaccuracies in the text displayed within the generated images. The capability to generate visual text is crucial, offering both academic interest and a wide range of practical applications. To produce accurate visual text images, state-of-the-art techniques adopt a glyph-controlled image generation approach, consisting of a text layout generator followed by an image generator that is conditioned on the generated text layout. Nevertheless, our study reveals that these models still face three primary challenges, prompting us to develop a testbed to facilitate future research. We introduce a benchmark, LenCom-Eval, specifically designed for testing models' capability in generating images with Lengthy and Complex visual text. Subsequently, we introduce a training-free framework to enhance the two-stage generation approaches. We examine the effectiveness of our approach on both LenCom-Eval and MARIO-Eval benchmarks and demonstrate notable improvements across a range of evaluation metrics, including CLIPScore, OCR precision, recall, F1 score, accuracy, and edit distance scores. For instance, our proposed framework improves the backbone model, TextDiffuser, by more than 23\% and 13.5\% in terms of OCR word F1 on LenCom-Eval and MARIO-Eval, respectively. Our work makes a unique contribution to the field by focusing on generating images with long and rare text sequences, a niche previously unexplored by existing literature

Figures (9)

• Figure 1: The proposed training-free framework to improve overall accuracy of visual text generation. This method consists of two main stages. First, it minimizes the overlapping of keyword bounding boxes created by the layout generator. Following that, OCR is employed to identify any spelling errors, following which we give a new mask region and the generated image to the pretrained in-painting image generation model. The second step is applied recursively.
• Figure 2: Accuracy of TextDiffuser across image subsets as the number of keywords increases: Performance drops as the number of words in the image increases.
• Figure 3: The generated bounding boxes and the corresponding glyph images with 6, 8, and 10 keywords: The overlapped area of the bounding boxes increases when the number of keywords increases.
• Figure 4: Examples of three frequent errors made by TextDiffuser: missing words (left), misspelling (middle), and blurry text (right).
• Figure 5: SA reduces the overlap of bounding boxes for keywords in the examples shown in Figure \ref{['fig:layout']}.