Language-Agnostic Visual Embeddings for Cross-Script Handwriting Retrieval

TL;DR

The paper addresses cross-script handwriting retrieval under severe stylistic and cross-lingual variation by introducing a lightweight asymmetric dual-encoder that maps handwritten images into a language-agnostic semantic space anchored by a frozen multilingual text encoder. It jointly optimizes instance-level alignment with $ abla ext{L}_{ITC}$ and semantic consistency via $ abla ext{L}_{INV}$, ensuring invariance to language and writing styles within a shared embedding space $ abla ext{V}$ and using $ au$ for contrastive losses. The approach demonstrates state-of-the-art or competitive performance on within-language benchmarks with far fewer parameters than large visual-language models, and shows robust cross-lingual retrieval, transfer from synthetic to real handwriting, and hardware-aware efficiency suitable for edge devices. The practical impact lies in enabling accurate, resource-efficient cross-script handwriting retrieval for large archives and multilingual document systems, without relying on expensive VLLMs.

Abstract

Handwritten word retrieval is vital for digital archives but remains challenging due to large handwriting variability and cross-lingual semantic gaps. While large vision-language models offer potential solutions, their prohibitive computational costs hinder practical edge deployment. To address this, we propose a lightweight asymmetric dual-encoder framework that learns unified, style-invariant visual embeddings. By jointly optimizing instance-level alignment and class-level semantic consistency, our approach anchors visual embeddings to language-agnostic semantic prototypes, enforcing invariance across scripts and writing styles. Experiments show that our method outperforms 28 baselines and achieves state-of-the-art accuracy on within-language retrieval benchmarks. We further conduct explicit cross-lingual retrieval, where the query language differs from the target language, to validate the effectiveness of the learned cross-lingual representations. Achieving strong performance with only a fraction of the parameters required by existing models, our framework enables accurate and resource-efficient cross-script handwriting retrieval.

Figures (5)

• Figure 1: Comparison of inference paradigms. Traditional OCR-based methods (top) suffer from irreversible error propagation on cursive handwriting (e.g., misrecognizing "from" as "Hom"). In contrast, our method (bottom) bypasses character decoding, achieving robust retrieval by directly aligning visual and semantic embeddings in a unified space.
• Figure 2: The overall architecture of the proposed framework. We employ an asymmetric dual-encoder design where a frozen multilingual text encoder acts as a semantic anchor generator, and a lightweight visual encoder learns to align handwritten images with these anchors. The training process follows a progressive strategy, evolving from synthetic pre-training to real-world fine-tuning. Two complementary objectives, Instance-Level Alignment ($\mathcal{L}_{ITC}$) and Semantic Consistency Alignment ($\mathcal{L}_{INV}$), are jointly optimized to establish a unified, semantic-invariant representation space. A hyperparameter $\lambda$ (default $\lambda=0.5$) is employed to balance local discriminative power and global structural compactness.
• Figure 3: Efficiency-Performance trade-off overview on the OOD set. (a) Accuracy@1 vs. Model Parameters. (b) Accuracy@1 vs. Inference Latency. Our framework achieves superior general retrieval performance while maintaining the minimum parameter count and lowest average latency.
• Figure 4: Latent space visualization via t-SNE. The top row reports the embedding distribution of GME-Qwen2VL-7B, while the bottom row illustrates the results of our method. The visualization involves multilingual samples, where CH, ENG, and SPA denote Chinese, English, and Spanish data, respectively. Cluster representation centers are represented by dark-colored sample points.
• Figure 5: Performance comparison between commercial GPU and our simulator. We report the trade-offs in accuracy, latency, and power. The blue arrow indicates accuracy drop due to quantization, while red arrows indicate significant improvements in efficiency.