Autoregressive Styled Text Image Generation, but Make it Reliable

TL;DR

Eruku reframes Handwritten Text Generation as a multimodal, prompt-conditioned autoregressive task that operates in a continuous VAE latent space. It introduces explicit visual-text synchronization tokens and a classifier-free guidance-like scheme to enforce strict content adherence while avoiding reliance on style image transcriptions. The model is trained in two phases on a large synthetic dataset and demonstrates robust generalization to unseen styles, arbitrary text length, and resilience to missing or noisy style inputs. Empirical results show improved style fidelity, readability, and long-context generation compared with state-of-the-art HTG approaches, with a simple, practical stopping mechanism and without external supervision networks.

Abstract

Generating faithful and readable styled text images (especially for Styled Handwritten Text generation - HTG) is an open problem with several possible applications across graphic design, document understanding, and image editing. A lot of research effort in this task is dedicated to developing strategies that reproduce the stylistic characteristics of a given writer, with promising results in terms of style fidelity and generalization achieved by the recently proposed Autoregressive Transformer paradigm for HTG. However, this method requires additional inputs, lacks a proper stop mechanism, and might end up in repetition loops, generating visual artifacts. In this work, we rethink the autoregressive formulation by framing HTG as a multimodal prompt-conditioned generation task, and tackle the content controllability issues by introducing special textual input tokens for better alignment with the visual ones. Moreover, we devise a Classifier-Free-Guidance-based strategy for our autoregressive model. Through extensive experimental validation, we demonstrate that our approach, dubbed Eruku, compared to previous solutions requires fewer inputs, generalizes better to unseen styles, and follows more faithfully the textual prompt, improving content adherence.

Figures (4)

• Figure 1: Our proposed Eruku model can generate text images with arbitrary length and with great text adherence, respecting both the generation text and with the conditioning writing style.
• Figure 2: Training framework of Eruku, our autoregressive text image generation model. We condition generation on: the textual content of a style image $T_s$ ("Style"), the generation text $T_g$ ("Gen"), and the style image $I_s$. Eruku is trained on next-token prediction, learning to generate an image containing the generation text $T_g$ with the same writing style as the style image $I_s$. Providing the style text $T_s$ enables the model to link each character with its representation style, but we also enable generation without the style text $T_s$ by dropping it during inference and using synchronization tokens to separate the sequence components. We represent images with VAE pippi2025zero continuous latents. Our model automatically learns to stop generation emitting the Visual End of Generation token$\texttt{<EOG>}$.
• Figure 3: Qualitative analysis of the CFG effect on generation at varying $\gamma$'s. We generate by giving as $I_s$ the top image in the figure (Style) and as $T_s$ the text contained in the bottom image (Ref.).
• Figure 4: Qualitative results between our proposed Eruku, and the State-of-the-Art Emuru and DiffPen models on the considered datasets. We task the models to generate a replica of the reported reference image (Ref.) by giving them as input the text contained in Ref. and the reported style image (Style).