Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Text Generation on Images with Synthetic Captions

Jun Young Koh, Sang Hyun Park, Joy Song

TL;DR

A low-cost approach is proposed by leveraging SDXL without any time-consuming training on large-scale datasets that employs a fine-tuning technique that examines the effects of data refinement levels and synthetic captions in generating legible text within the image.

Abstract

The recent emergence of latent diffusion models such as SDXL and SD 1.5 has shown significant capability in generating highly detailed and realistic images. Despite their remarkable ability to produce images, generating accurate text within images still remains a challenging task. In this paper, we examine the validity of fine-tuning approaches in generating legible text within the image. We propose a low-cost approach by leveraging SDXL without any time-consuming training on large-scale datasets. The proposed strategy employs a fine-tuning technique that examines the effects of data refinement levels and synthetic captions. Moreover, our results demonstrate how our small scale fine-tuning approach can improve the accuracy of text generation in different scenarios without the need of additional multimodal encoders. Our experiments show that with the addition of random letters to our raw dataset, our model's performance improves in producing well-formed visual text.

Improving Text Generation on Images with Synthetic Captions

TL;DR

A low-cost approach is proposed by leveraging SDXL without any time-consuming training on large-scale datasets that employs a fine-tuning technique that examines the effects of data refinement levels and synthetic captions in generating legible text within the image.

Abstract

The recent emergence of latent diffusion models such as SDXL and SD 1.5 has shown significant capability in generating highly detailed and realistic images. Despite their remarkable ability to produce images, generating accurate text within images still remains a challenging task. In this paper, we examine the validity of fine-tuning approaches in generating legible text within the image. We propose a low-cost approach by leveraging SDXL without any time-consuming training on large-scale datasets. The proposed strategy employs a fine-tuning technique that examines the effects of data refinement levels and synthetic captions. Moreover, our results demonstrate how our small scale fine-tuning approach can improve the accuracy of text generation in different scenarios without the need of additional multimodal encoders. Our experiments show that with the addition of random letters to our raw dataset, our model's performance improves in producing well-formed visual text.
Paper Structure (19 sections, 12 figures, 2 tables)

This paper contains 19 sections, 12 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-image generation
  4. Synthetic data
  5. Method
  6. Data preparation
  7. Mixture of base dataset and synthetic dataset
  8. Model training
  9. Evaluation
  10. Human evaluation
  11. CER metrics
  12. CLIP score
  13. TIFA score
  14. Discussion
  15. Conclusion
  16. ...and 4 more sections

Figures (12)

  • Figure 1: Comparison of visual results of our model and different text-to-image models. The best results from 16 generated images were selected.
  • Figure 2: Comparison of the performance of our model trained on various blending ratios of refined captions and automatic captions. Only the model trained on CogVLM captioning dataset fails to reconstruct the 'aqua' text, whereas models trained with higher percentage of refined data render text more accurately.
  • Figure 3: Heat map comparing the performance of our different models through human evaluation results. We provide a legend (right) that shows the detailed data composition for each model (m0 to m9) in terms of refinement level and type of synthetic data.
  • Figure 4: Dataset consisting of raw data from Danbooru as well as synthetic data. This image shows sample raw data from Danbooru(left), sample synthetic data for real words (center) and sample synthetic data composed of random combination of letters (right).
  • Figure 5: The inference result of model when synthetic dataset is exclusively used. Model fails to follow the prompt, especially related to background. The prompt used for inference is "A girl with long red hair and red eyes holds a sunflower amidst a whirlwind of signs, showcasing a dreamy and surreal atmosphere. The sign has a English text with 'MINIMUM SPEED 45'." "A girl with pink hair and a purple top hat is holding a sign that reads 'Conun Drum', with a surprised expression on her face. The image has an artistic and whimsical feel, with the girl's attire and the background design adding to the fantasy theme. "A young male with black hair and red eyes is holding a phone, surrounded by a dynamic environment with a road sign and a sign. With the text 'Happy Birthday IZAYA'."
  • ...and 7 more figures