Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WriteViT: Handwritten Text Generation with Vision Transformer

Dang Hoai Nam, Huynh Tong Dang Khoa, Vo Nguyen Le Duy

TL;DR

WriteViT addresses one-shot handwritten text generation by integrating Vision Transformers across Generator, Recognizer, and Writer Identifier, coupled with a multi-scale architecture and Conditional Positional Encoding. The method achieves high-fidelity, style-consistent handwriting for both English and Vietnamese, improves downstream HTR when used for data augmentation, and demonstrates strong generalization across vocabulary and unseen writer styles. Key contributions include a ViT-based, multi-scale generator; ViT-based recognizer and writer encoder; and a training scheme balancing adversarial and recognition objectives. The results show competitive visual quality (low $FID$/$KID$) and improved recognition metrics, highlighting the practicality of Transformer-based handwriting synthesis for multilingual, low-resource scenarios and suggesting avenues for personalization and broader language coverage.

Abstract

Humans can quickly generalize handwriting styles from a single example by intuitively separating content from style. Machines, however, struggle with this task, especially in low-data settings, often missing subtle spatial and stylistic cues. Motivated by this gap, we introduce WriteViT, a one-shot handwritten text synthesis framework that incorporates Vision Transformers (ViT), a family of models that have shown strong performance across various computer vision tasks. WriteViT integrates a ViT-based Writer Identifier for extracting style embeddings, a multi-scale generator built with Transformer encoder-decoder blocks enhanced by conditional positional encoding (CPE), and a lightweight ViT-based recognizer. While previous methods typically rely on CNNs or CRNNs, our design leverages transformers in key components to better capture both fine-grained stroke details and higher-level style information. Although handwritten text synthesis has been widely explored, its application to Vietnamese -- a language rich in diacritics and complex typography -- remains limited. Experiments on Vietnamese and English datasets demonstrate that WriteViT produces high-quality, style-consistent handwriting while maintaining strong recognition performance in low-resource scenarios. These results highlight the promise of transformer-based designs for multilingual handwriting generation and efficient style adaptation.

WriteViT: Handwritten Text Generation with Vision Transformer

TL;DR

WriteViT addresses one-shot handwritten text generation by integrating Vision Transformers across Generator, Recognizer, and Writer Identifier, coupled with a multi-scale architecture and Conditional Positional Encoding. The method achieves high-fidelity, style-consistent handwriting for both English and Vietnamese, improves downstream HTR when used for data augmentation, and demonstrates strong generalization across vocabulary and unseen writer styles. Key contributions include a ViT-based, multi-scale generator; ViT-based recognizer and writer encoder; and a training scheme balancing adversarial and recognition objectives. The results show competitive visual quality (low /) and improved recognition metrics, highlighting the practicality of Transformer-based handwriting synthesis for multilingual, low-resource scenarios and suggesting avenues for personalization and broader language coverage.

Abstract

Humans can quickly generalize handwriting styles from a single example by intuitively separating content from style. Machines, however, struggle with this task, especially in low-data settings, often missing subtle spatial and stylistic cues. Motivated by this gap, we introduce WriteViT, a one-shot handwritten text synthesis framework that incorporates Vision Transformers (ViT), a family of models that have shown strong performance across various computer vision tasks. WriteViT integrates a ViT-based Writer Identifier for extracting style embeddings, a multi-scale generator built with Transformer encoder-decoder blocks enhanced by conditional positional encoding (CPE), and a lightweight ViT-based recognizer. While previous methods typically rely on CNNs or CRNNs, our design leverages transformers in key components to better capture both fine-grained stroke details and higher-level style information. Although handwritten text synthesis has been widely explored, its application to Vietnamese -- a language rich in diacritics and complex typography -- remains limited. Experiments on Vietnamese and English datasets demonstrate that WriteViT produces high-quality, style-consistent handwriting while maintaining strong recognition performance in low-resource scenarios. These results highlight the promise of transformer-based designs for multilingual handwriting generation and efficient style adaptation.
Paper Structure (26 sections, 5 equations, 4 figures, 6 tables)

This paper contains 26 sections, 5 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Early Methods with Recurrent Neural Networks
  4. Generative Approaches
  5. Proposed Approach
  6. Motivation
  7. Problem Formulation
  8. Overall Architecture
  9. Generator
  10. Discriminator
  11. Recognizer
  12. Writer Identifier
  13. Objective Functions
  14. Experiments
  15. Image Quality Assessment
  16. ...and 11 more sections

Figures (4)

  • Figure 1: Overview of the proposed WriteViT architecture.
  • Figure 2: Qualitative comparison of generated handwriting from different models given the same style and content inputs. Each row represents a different model, while each column corresponds to a specific input sentence.
  • Figure 3: Qualitative reconstruction results. Each row corresponds to a different model, conditioned on the same ground-truth style (bottom row) and target text. The goal is to reproduce the target content while preserving the handwriting style.
  • Figure 4: Qualitative comparison of Vietnamese handwriting generation on the VNOnDB dataset. Each model is conditioned on the same input text and writer style.