One-Shot Diffusion Mimicker for Handwritten Text Generation

TL;DR

A One-shot Diffusion Mimicker (One-DM) to generate handwritten text that can mimic any calligraphic style with only one reference sample, and develops a novel style-enhanced module to improve the style extraction by incorporating high-frequency components from a single sample.

Abstract

Existing handwritten text generation methods often require more than ten handwriting samples as style references. However, in practical applications, users tend to prefer a handwriting generation model that operates with just a single reference sample for its convenience and efficiency. This approach, known as "one-shot generation", significantly simplifies the process but poses a significant challenge due to the difficulty of accurately capturing a writer's style from a single sample, especially when extracting fine details from the characters' edges amidst sparse foreground and undesired background noise. To address this problem, we propose a One-shot Diffusion Mimicker (One-DM) to generate handwritten text that can mimic any calligraphic style with only one reference sample. Inspired by the fact that high-frequency information of the individual sample often contains distinct style patterns (e.g., character slant and letter joining), we develop a novel style-enhanced module to improve the style extraction by incorporating high-frequency components from a single sample. We then fuse the style features with the text content as a merged condition for guiding the diffusion model to produce high-quality handwritten text images. Extensive experiments demonstrate that our method can successfully generate handwriting scripts with just one sample reference in multiple languages, even outperforming previous methods using over ten samples. Our source code is available at https://github.com/dailenson/One-DM.

Figures (25)

• Figure 1: User experience comparisons between one-shot and few-shot handwritten text generation methods. It reveals that one-shot setting leads to a better user experience.
• Figure 2: Handwritten text samples and corresponding high-frequency components. We find that high-frequency components have more pronounced character contours, clearly showcasing the style patterns, such as character slant and cursive connections.
• Figure 3: Overview of the proposed method. The style reference initially passes through a high-pass filter to extract its high-frequency components. Subsequently, the spatial and the high-frequency style encoders independently extract style features $F_{spa}$ and $F_{fre}$ from the style reference and its high-frequency information, respectively. $F_{spa}$, after being filtered through a gate mechanism, is fused with $F_{fre}$ and content features $E$ in the fusion module. The merged feature then serves as a condition input to guide the diffusion generation process.
• Figure 4: Qualitative comparisons between our method with state-of-the-art methods on handwritten text generation with both specific textual content and desired handwriting style in the IAM dataset. We utilize the identical guiding text, 'The greatest test of courage on earth is to bear defeat without losing heart,' across all handwriting generation methods, directing them to produce text in varied styles. Better zoom in 200%. $\ast$ denotes one-shot methods, while others are few-shot methods.
• Figure 5: Each row shows results from our One-DM and few-shot methods on IAM dataset; readers are invited to identify our method. The answer is at the paper's end.