ScriptViT: Vision Transformer-Based Personalized Handwriting Generation
Sajjan Acharya, Rajendra Baskota
TL;DR
ScriptViT addresses the challenge of personalized offline handwriting generation by capturing global writer-specific styles and long-range structural patterns. It introduces a Vision Transformer-based Style Image Encoder to construct a global Style Memory, and a Content Query Encoder that feeds a Transformer-based Style-Content Fusion Core through cross-attention, enabling faithful style-content integration. An interpretability module, Salient Stroke Attention Analysis (SSAA), reveals stroke-level contributions to style transfer by projecting decoder attention onto ink regions. On the IAM dataset, ScriptViT delivers improved Handwriting Distance ($+21.39\%$) and Kernel Inception Distance ($+9.42\%$) while maintaining competitive Fréchet Inception Distance, demonstrating both stylistic fidelity and content accuracy. The combination of global style encoding, cross-attention fusion, and SSAA provides a practical, interpretable framework for high-quality, personalized handwriting synthesis with potential applications in data augmentation and document restoration. The architecture is summarized by $N=5$ style images, a Transformer decoder with $N_x=3$ layers, $H=8$ attention heads, and embedding dimension $D_{model}=512$, highlighting a compact yet effective design for writer-specific handwriting generation.
Abstract
Styled handwriting generation aims to synthesize handwritten text that looks both realistic and aligned with a specific writer's style. While recent approaches involving GAN, transformer and diffusion-based models have made progress, they often struggle to capture the full spectrum of writer-specific attributes, particularly global stylistic patterns that span long-range spatial dependencies. As a result, capturing subtle writer-specific traits such as consistent slant, curvature or stroke pressure, while keeping the generated text accurate is still an open problem. In this work, we present a unified framework designed to address these limitations. We introduce a Vision Transformer-based style encoder that learns global stylistic patterns from multiple reference images, allowing the model to better represent long-range structural characteristics of handwriting. We then integrate these style cues with the target text using a cross-attention mechanism, enabling the system to produce handwritten images that more faithfully reflect the intended style. To make the process more interpretable, we utilize Salient Stroke Attention Analysis (SSAA), which reveals the stroke-level features the model focuses on during style transfer. Together, these components lead to handwriting synthesis that is not only more stylistically coherent, but also easier to understand and analyze.