CalliRewrite: Recovering Handwriting Behaviors from Calligraphy Images without Supervision

TL;DR

CalliRewrite tackles the challenge of recovering handwriting behaviors from calligraphy images in an unsupervised, cross-tool setting. It presents a coarse-to-fine pipeline that first extracts stroke sequences with a CNN-LSTM under novel unsupervised losses and progressive training, then finely tunes tool-specific trajectories via SAC-based reinforcement learning in a CalliEnv framework. The approach yields faithful replays across languages (Chinese, English, Ancient Egyptian, Tamil) in both simulation and real robot experiments, evaluated with Stroke Number Ratio and Chamfer Distance. The results show superior performance to other unsupervised methods and competitive results with supervised baselines, highlighting the potential of unsupervised, tool-aware robotic handwriting for diverse scripts.

Abstract

Human-like planning skills and dexterous manipulation have long posed challenges in the fields of robotics and artificial intelligence (AI). The task of reinterpreting calligraphy presents a formidable challenge, as it involves the decomposition of strokes and dexterous utensil control. Previous efforts have primarily focused on supervised learning of a single instrument, limiting the performance of robots in the realm of cross-domain text replication. To address these challenges, we propose CalliRewrite: a coarse-to-fine approach for robot arms to discover and recover plausible writing orders from diverse calligraphy images without requiring labeled demonstrations. Our model achieves fine-grained control of various writing utensils. Specifically, an unsupervised image-to-sequence model decomposes a given calligraphy glyph to obtain a coarse stroke sequence. Using an RL algorithm, a simulated brush is fine-tuned to generate stylized trajectories for robotic arm control. Evaluation in simulation and physical robot scenarios reveals that our method successfully replicates unseen fonts and styles while achieving integrity in unknown characters.

Figures (12)

• Figure 1: Our method, CalliRewrite, enables unsupervised and tool-dependent robot calligraphy. During training, we provide only glyph images and virtual writing utensils. Our two-stage model gradually proposes and fine-tunes applicable writing trajectories conditioned on specific tools. In validation, we apply the parametric fine control on the Dobot robotic arm, which accurately redraws the characters.
• Figure 2: Method overview: (a) Generating coarse writing trajectories using unsupervised LSTM with tailored loss for human-like glyph decomposition. (b) Reinforcement learning with utensil properties for fine-tuning dexterous control.
• Figure 3: Brief introduction of our modeled writing utensils. More information can be found in the supplemental material.
• Figure 4: (a) Progressive Training ("pt") Comparison. (b) Impact of Fine-Tuning ("ft") on Stroke Connectivity and Tool-Aware Control, including joining disconnected strokes and adding brush-related control at both ends of the strokes.
• Figure 5: (a) Visualization on decomposed coarse sequences. From blue to red: stroke order from the 1st to the 100th. (b) Tool-aware fine-tuning with three writing utensils.