Zero-Shot Styled Text Image Generation, but Make It Autoregressive

TL;DR

Emuru tackles the challenge of generating styled text images that generalize to unseen handwriting and font styles while supporting arbitrarily long outputs. It presents a two-stage autoregressive framework: a beta-VAE learns a compact, background-free latent representation of styled text lines, and a Transformer generates a sequence of latent embeddings conditioned on style and content, with final rendering via the VAE Decoder. Trained on a large synthetic dataset with over 2.2 million samples from 100k fonts, Emuru demonstrates strong zero-shot generalization across IAM, CVL, RIMES, and a diverse Karaoke dataset, while producing outputs with reduced background artifacts. This approach enables flexible, background-free text image generation suitable for editing and downstream document-analysis tasks, representing the first autoregressive solution for offline styled HTG with broad style generalization.

Abstract

Styled Handwritten Text Generation (HTG) has recently received attention from the computer vision and document analysis communities, which have developed several solutions, either GAN- or diffusion-based, that achieved promising results. Nonetheless, these strategies fail to generalize to novel styles and have technical constraints, particularly in terms of maximum output length and training efficiency. To overcome these limitations, in this work, we propose a novel framework for text image generation, dubbed Emuru. Our approach leverages a powerful text image representation model (a variational autoencoder) combined with an autoregressive Transformer. Our approach enables the generation of styled text images conditioned on textual content and style examples, such as specific fonts or handwriting styles. We train our model solely on a diverse, synthetic dataset of English text rendered in over 100,000 typewritten and calligraphy fonts, which gives it the capability to reproduce unseen styles (both fonts and users' handwriting) in zero-shot. To the best of our knowledge, Emuru is the first autoregressive model for HTG, and the first designed specifically for generalization to novel styles. Moreover, our model generates images without background artifacts, which are easier to use for downstream applications. Extensive evaluation on both typewritten and handwritten, any-length text image generation scenarios demonstrates the effectiveness of our approach.

Figures (17)

• Figure 1: Our proposed Emuru model can generate images of any-length text lines mimicking out-of-distribution handwriting styles.
• Figure 2: Our Emuru approach consists of a VAE (a) and an autoregressive Transformer Encoder-Decoder (b), both trained on a massive synthetic dataset of texts rendered in different fonts. At inference time (c), Emuru is given a reference style image, the text in the reference style image, and the desired text and is tasked to iteratively generate the output styled image, autonomously deciding when to stop.
• Figure 3: Characters and padding VAE embeddings distribution obtained via t-SNE.
• Figure 4: Qualitative comparison between Emuru, the GAN-based VATr++, and the Diffusion Model-based DiffPen when generating images from the considered datasets. We report the input style image used for guiding the generation and another reference image in the same style. We let the models generate the same text as in the reference to better observe the style imitation capabilities of the models.
• Figure 5: Results of image generation from a style image with background artifacts obtained with Emuru and DiffPen. Thanks to the VAE training, Emuru images do not reproduce these artifacts.