Beyond Patches: Global-aware Autoregressive Model for Multimodal Few-Shot Font Generation

TL;DR

GAR-Font tackles the global coherence gap in autoregressive few-shot font generation by introducing a global-aware tokenizer (G-Tok) that fuses local stroke details with global stylistic cues, paired with an autoregressive generator and a multimodal style encoder that incorporates lightweight language guidance. The model is trained in two stages (visual pretraining, followed by vision–language adaptation) and refined with a post-refinement pipeline (NFA and SE) to boost both structural fidelity and stylistic coherence. Empirical results show GAR-Font outperforms prior vision-only FFG methods on UFSC/UFUC benchmarks, with multimodal variants achieving further gains using textual prompts with fewer visual references. The work demonstrates the practicality of text-informed, global-context font synthesis and provides scalable paths for high-quality, controllable font generation in logographic scripts.

Abstract

Manual font design is an intricate process that transforms a stylistic visual concept into a coherent glyph set. This challenge persists in automated Few-shot Font Generation (FFG), where models often struggle to preserve both the structural integrity and stylistic fidelity from limited references. While autoregressive (AR) models have demonstrated impressive generative capabilities, their application to FFG is constrained by conventional patch-level tokenization, which neglects global dependencies crucial for coherent font synthesis. Moreover, existing FFG methods remain within the image-to-image paradigm, relying solely on visual references and overlooking the role of language in conveying stylistic intent during font design. To address these limitations, we propose GAR-Font, a novel AR framework for multimodal few-shot font generation. GAR-Font introduces a global-aware tokenizer that effectively captures both local structures and global stylistic patterns, a multimodal style encoder offering flexible style control through a lightweight language-style adapter without requiring intensive multimodal pretraining, and a post-refinement pipeline that further enhances structural fidelity and style coherence. Extensive experiments show that GAR-Font outperforms existing FFG methods, excelling in maintaining global style faithfulness and achieving higher-quality results with textual stylistic guidance.

Figures (20)

• Figure 1: GAR-Font results under visual and multimodal few-shot settings. The generated poem reflects the key contributions of our model: global-aware tokenization for style fidelity, multimodal style encoding for text-image control, reduced reference requirements, and an autoregressive design that enables controllable high-quality font synthesis.
• Figure 2: The overall architecture of GAR-Font. It comprises a global-aware tokenizer (G-Tok), and an AR generator, equipped with a multimodal style encoder.
• Figure 3: (a) Overview of the G-Tok architecture, which adopts a hybrid CNN–ViT design. (b) Details of the global ViT encoder and causal ViT decoder. With self- and causal-attention, G-Tok captures global dependencies, enabling coherent and high-quality font synthesis.
• Figure 4: GAR-Font adopts a two-stage training: (a) Visual Pretraining builds a stable content–style space via token and pixel losses; (b) Vision–language Adaptation aligns text embeddings with style features with a lightweight adapter for text-guided generation while preserving visual priors.
• Figure 5: Qualitative results on vision-only FFG (UFSC, Large dataset). / indicate structural errors and style mismatches. GAR-Font($I_8$, +NFA+SE) produces the most faithful glyphs with superior structure and style alignment.