MENTOR: Multilingual tExt detectioN TOward leaRning by analogy

TL;DR

This paper addresses the challenge of detecting text in multilingual scene images, including unseen scripts, without collecting new labeled data or retraining. It introduces MENTOR, a three-component framework that combines zero-cost printed-text external information with a meta-mapping to language-specific kernels for detection-by-matching, enabling open-set language detection. The Dynamic Guide, Text Finder, and Language Mapper modules are trained with a two-stage process and augmented data to generalize to unseen languages, achieving competitive results against supervised baselines on multilingual benchmarks and unseen Malayalam. The approach has practical implications for autonomous systems operating in diverse linguistic environments by reducing data collection and retraining costs while maintaining robust multilingual text detection capabilities.

Abstract

Text detection is frequently used in vision-based mobile robots when they need to interpret texts in their surroundings to perform a given task. For instance, delivery robots in multilingual cities need to be capable of doing multilingual text detection so that the robots can read traffic signs and road markings. Moreover, the target languages change from region to region, implying the need of efficiently re-training the models to recognize the novel/new languages. However, collecting and labeling training data for novel languages are cumbersome, and the efforts to re-train an existing/trained text detector are considerable. Even worse, such a routine would repeat whenever a novel language appears. This motivates us to propose a new problem setting for tackling the aforementioned challenges in a more efficient way: "We ask for a generalizable multilingual text detection framework to detect and identify both seen and unseen language regions inside scene images without the requirement of collecting supervised training data for unseen languages as well as model re-training". To this end, we propose "MENTOR", the first work to realize a learning strategy between zero-shot learning and few-shot learning for multilingual scene text detection.

Figures (3)

• Figure 1: Left: The detection results of an existing detector for multilingual scene text could be problematic when the model encounters a language that hasn’t been learned previously. Upper-right: For understanding unseen languages, previous approaches often require large-scale datasets collected and annotated by humans to retain the models for recognizing new languages. Lower-right: Free-of-charge printed text images are the only data required for our method to detect texts of unseen languages without any effort on model re-training.
• Figure 2: Overview of our proposed framework "MENTOR". Assuming that the target language is Japanese, we will prepare 5 sheets with printed Japanese texts (which are almost zero-cost) as the input of Dynamic Guide (DG) while the scene text image to be examined is the input of our Text Finder (TF). DG extracts the Japanese text attributes as auxiliary information to the prediction. TF is split into two paths after the backbone extraction. The Gray block is a language-agnostic detection sub-branch that is trained separately and outputs the centerness map and the text map, $Mask_{text}$. The lower sub-branch of TF predicts scene text language, and the middle feature $F_{mid}$ will multiply with $Mask_{text}$ to send the text-only areas to the Language Mapper (LM). In the Language Mapper, the transferred Japanese text attributes are used as the weights of different dynamic kernels, and convoluted with the scene text features. Through progressive comparison, the regions where the scene text and the printed text have similar language characteristics are selected. Later, the Japanese detection result is obtained. The detection flow can be directly applied to detect other unseen languages without retraining the model.
• Figure 3: (a) Detection results trained with disjoint synthetic scene data. The target language from left to right is Japanese, Japanese, and Korean. All the real scene text is detected regardless of its language, which reveals that a good matching relationship is not well-learned due to the insufficient training of the language mapper. (b) The model was trained on intersection synthetic data. Korean is the unseen language. The detector can detect both synthesized and real scene texts in Korean without being confused with other languages.