Textured Word-As-Image illustration

TL;DR

The paper tackles semantic typography by presenting an automatic pipeline that reshapes a selected letter to reflect a semantic concept and textures the letter via diffusion-based generation, ensuring readability. It integrates Word-as-Illustration for letter deformation with Stable Diffusion XL for concept-driven textures, followed by a precise texture-mapping stage using alpha masking and Lanczos resampling. A Gradio-based UI enables real-time texture scale and background color adjustments, and a user study demonstrates that textures convey semantic meaning without significantly compromising legibility. While effective, the approach depends on pretrained diffusion models and acknowledges limitations for abstract concepts, suggesting future model retraining or fine-tuning to broaden coverage.

Abstract

In this paper, we propose a novel fully automatic pipeline to generate text images that are legible and strongly aligned to the desired semantic concept taken from the users' inputs. In our method, users are able to put three inputs into the system, including a semantic concept, a word, and a letter. The semantic concept will be used to change the shape of the input letter and generate the texture based on the pre-defined prompt using stable diffusion models. Our pipeline maps the texture on a text image in a way that preserves the readability of the whole output while preserving legibility. The system also provides real-time adjustments for the user to change the scale of the texture and apply it to the text image. User evaluations demonstrate that our method effectively represents semantic meaning without compromising legibility, making it a robust and innovative tool for graphic design, logo creation, and artistic typography.

Figures (4)

• Figure 1: The Proposed pipeline takes three inputs including "Semantic Concept", "Word", and a "Letter". In the provided example the inputs are "TREE" for semantic concept and "NATURE" for word choice, and the user has chosen letter "T" as the input. Then these inputs are sent to the "Word-As-Image" moduleiluz2023word to produce a text image with the stylized letter chosen by the user. Also, the "Semantic Concept" will be used to generate a subtle texture using a stable diffusion modelpodell2023sdxl. For instance, the "Word-As-Image" module has produced a word "NATURE" and changed the letter "T" into tree. Also, the stable diffusion module has produced a pattern of trees. Then, the generated texture will be mapped to the text image using the developed "Texture Mapping" module to generate an initial output. After output generation, the user is able to use texture scaling to better fit the texture and the color picker to change the default color of the background.
• Figure 2: Final outputs of the proposed pipeline based on semantic concepts, target words, and stylized letters using ”Word as Illustration” mode iluz2023word. (a) Uses ”FLOWER” as both the semantic concept and the word, with the letter ”O” stylized via . (b) Uses ”BUNNY” as the semantic concept and as the word, with the letter ”Y” stylized.
• Figure 3: Distribution ratings for how well the generated texture conveys the semantic concepts of the final generated outputs. As it is show, roughly 65% of the testers have indicated strong alignment between the textures and semantic concepts.
• Figure 4: 66.7 percent of the participants have stated that texture of these two images better represented the meaning of semantic concept.