VGTS: Visually Guided Text Spotting for Novel Categories in Historical Manuscripts

TL;DR

This work tackles open-set, low-resource text spotting in historical manuscripts by introducing VGTS, a framework that guides spotting with a single support example. It combines dense correlation matching with a spatial alignment module featuring Dual Spatial Attention and Geometric Matching to localize novel characters without fine-tuning, and it employs a torus loss to improve distance-metric discrimination under data scarcity. The approach is validated on a newly introduced Dongba dataset and multiple public datasets, where VGTS consistently outperforms baselines in mAP, recall, and F1, and shows robustness to rotation, long-tail distributions, and degraded support images. The method promises practical utility for historians by enabling rapid cataloging of novel symbols with minimal annotation and preparation effort.

Abstract

In the field of historical manuscript research, scholars frequently encounter novel symbols in ancient texts, investing considerable effort in their identification and documentation. Although existing object detection methods achieve impressive performance on known categories, they struggle to recognize novel symbols without retraining. To address this limitation, we propose a Visually Guided Text Spotting (VGTS) approach that accurately spots novel characters using just one annotated support sample. The core of VGTS is a spatial alignment module consisting of a Dual Spatial Attention (DSA) block and a Geometric Matching (GM) block. The DSA block aims to identify, focus on, and learn discriminative spatial regions in the support and query images, mimicking the human visual spotting process. It first refines the support image by analyzing inter-channel relationships to identify critical areas, and then refines the query image by focusing on informative key points. The GM block, on the other hand, establishes the spatial correspondence between the two images, enabling accurate localization of the target character in the query image. To tackle the example imbalance problem in low-resource spotting tasks, we develop a novel torus loss function that enhances the discriminative power of the embedding space for distance metric learning. To further validate our approach, we introduce a new dataset featuring ancient Dongba hieroglyphics (DBH) associated with the Naxi minority of China. Extensive experiments on the DBH dataset and other public datasets, including EGY, VML-HD, TKH, and NC, show that VGTS consistently surpasses state-of-the-art methods. The proposed framework exhibits great potential for application in historical manuscript text spotting, enabling scholars to efficiently identify and document novel symbols with minimal annotation effort.

Figures (14)

• Figure 1: The overall framework of VGTS. The proposed framework can spot novel interested character categories conditioned with one support image. Based on the feature extracted by the backbone network, the correlation matching module computes the correlation map to match the pair of individual feature maps, while the spatial alignment module predicts the localization boxes and spotting results.
• Figure 2: Correlation map computation using pairs of individual features involves first extracting image descriptors $f_{s}$ and $f_{q}$ from images $I_{s}$ and $I_{q}$, respectively. Subsequently, all pairs of individual feature matches, $f_{s}^{kl}$ and $f_{q}^{ab}$, are represented in the 4D space of matches $(a,b,k,l)$, where the matching score is stored in a 4D correlation tensor. This 4D tensor is then reshaped into a 3D tensor with dimensions $h_{q}$, $w_{q}$, and $(h_{s}\times w_{s})$, allowing for the construction of a correlation map. At a specific spatial location $(a,b)$, the correlation map $C_{r}$ provides an aggregation of all similarities between $f_{q}(a,b)$ and all $f_{s}$.
• Figure 3: Flowchart of dual spatial attention block. Given the correlation map $C_{r}$ from the correlation matching module, the first step is to attention to the dimension $d = (h_{s} \times w_{s})$ of the correlation map, which can be regarded as a refinement for the support image. The next step is to attention to the dimensions $w_{q}$ and $h_{q}$ of the correlation map, which can be regarded as a refinement for the query image.
• Figure 4: Data distribution analysis. (a) Prior to training, the data may be distributed in a complex and nonlinear fashion, with no clear separation between positive and negative examples. (b) During training, the model may be able to identify some patterns and achieve a degree of separation between positive and negative examples. Some examples may still remain in the margin gap, where the decision boundary is unclear and further optimization is needed to achieve greater separation. (c) To achieve optimal performance, we expect the values of positive examples to be greater than the margin $m_{pos}$, while the values of negative examples should be less than the margin $m_{neg}$.
• Figure 5: Character frequency distribution in the TKH dataset and comparative mAP scores of different methods.