SceneVTG++: Controllable Multilingual Visual Text Generation in the Wild

TL;DR

SceneVTG++ tackles the challenge of realistic multilingual visual text in natural scenes by presenting a two-stage framework that separates text layout/content generation (TLCG) from pixel-level text rendering (CLTD). TLCG leverages multimodal large models to propose contextually appropriate text regions and multilingual content, while CLTD uses a conditioned diffusion process to render text that faithfully integrates with backgrounds and adheres to explicit attributes. The authors introduce SceneVTG-Erase++ and SceneVTG-Syn datasets to support training and evaluation, achieving state-of-the-art fidelity, reasonability, utility, and controllability in both English and multilingual settings. The work advances practical OCR data synthesis for robust detection and recognition, with strong potential to improve real-world scene-text understanding applications.

Abstract

Generating visual text in natural scene images is a challenging task with many unsolved problems. Different from generating text on artificially designed images (such as posters, covers, cartoons, etc.), the text in natural scene images needs to meet the following four key criteria: (1) Fidelity: the generated text should appear as realistic as a photograph and be completely accurate, with no errors in any of the strokes. (2) Reasonability: the text should be generated on reasonable carrier areas (such as boards, signs, walls, etc.), and the generated text content should also be relevant to the scene. (3) Utility: the generated text can facilitate to the training of natural scene OCR (Optical Character Recognition) tasks. (4) Controllability: The attribute of the text (such as font and color) should be controllable as needed. In this paper, we propose a two stage method, SceneVTG++, which simultaneously satisfies the four aspects mentioned above. SceneVTG++ consists of a Text Layout and Content Generator (TLCG) and a Controllable Local Text Diffusion (CLTD). The former utilizes the world knowledge of multi modal large language models to find reasonable text areas and recommend text content according to the nature scene background images, while the latter generates controllable multilingual text based on the diffusion model. Through extensive experiments, we respectively verified the effectiveness of TLCG and CLTD, and demonstrated the state-of-the-art text generation performance of SceneVTG++. In addition, the generated images have superior utility in OCR tasks like text detection and text recognition. Codes and datasets will be available.

Figures (12)

• Figure 1: In terms of fidelity, reasonability, utility, and controllability, our proposed SceneVTG++ excels in adequately satisfying all four criteria when compared with various other methods. Zoom in for better views.
• Figure 2: Comparison of pipelines for different visual text generation methods
• Figure 3: The overall pipeline of SceneVTG++. With background images and predefined text prompts as input, TLCG generates reasonable text layouts and recommends appropriate text contents. CLTD then generates text on the background image based on TLCG outputs along with some other text attributes.
• Figure 4: An example of TLCG workflow that generate reasonable text layout and content in two steps. The initial step involves identifying proposal points and contents of the text, followed by generating the text box in the next step.
• Figure 5: The detailed architecture of the Controllable Local Text Diffusion (CLTD). Taking the text layouts, contents, attributes and background images as input, CLTD renders desired text upon the background image.