Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control

Qisheng Liao, Liang Li, Yulang Fei, Gus Xia

TL;DR

CalliffusionV2 is a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control that excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach.

Abstract

In this paper, we introduce CalliffusionV2, a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control. Unlike previous approaches that rely solely on image or text inputs and lack fine-grained control, our system leverages both images to guide generations at fine-grained levels and natural language texts to describe the features of generations. CalliffusionV2 excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach. It is also capable of generating non-Chinese characters without prior training. Comprehensive tests confirm that our system produces calligraphy that is both stylistically accurate and recognizable by neural network classifiers and human evaluators.

CalliffusionV2: Personalized Natural Calligraphy Generation with Flexible Multi-modal Control

TL;DR

CalliffusionV2 is a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control that excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach.

Abstract

In this paper, we introduce CalliffusionV2, a novel system designed to produce natural Chinese calligraphy with flexible multi-modal control. Unlike previous approaches that rely solely on image or text inputs and lack fine-grained control, our system leverages both images to guide generations at fine-grained levels and natural language texts to describe the features of generations. CalliffusionV2 excels at creating a broad range of characters and can quickly learn new styles through a few-shot learning approach. It is also capable of generating non-Chinese characters without prior training. Comprehensive tests confirm that our system produces calligraphy that is both stylistically accurate and recognizable by neural network classifiers and human evaluators.
Paper Structure (22 sections, 4 equations, 7 figures, 3 tables)

This paper contains 22 sections, 4 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chinese Font and Calligraphy Generation
  4. Diffusion Models with Fine-tuning
  5. Methods
  6. Overview
  7. Preliminaries
  8. Training
  9. Inference
  10. New Style Generation via Few-shot Fine-tuning
  11. Experiments
  12. Experiment Setups
  13. Generations
  14. General Use Case
  15. Few-shots Fine-tuning
  16. ...and 7 more sections

Figures (7)

  • Figure 1: A Chinese calligraphy artwork generated by our system. The original generations are in black with white backgrounds, we modify the size and color manually.
  • Figure 2: The full architecture of our natural Chinese calligraphy generation system.
  • Figure 3: General use case of our system in CalliffusionV2-pro mode with different types of input images and prompts. The generations are based on prompts and skeleton images that are converted by input images.
  • Figure 4: Fine-tuning new style generations. Two new calligraphy styles and two new digital fonts are tested with 5 shots each.
  • Figure 5: Generations with fine-grained modifications. The input images with highlight parts are changed with different operations which successfully affects the generations
  • ...and 2 more figures