The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection

TL;DR

The paper identifies two key reasons for the gap between academic benchmarks and real-world scene text detection: a Fine-tuning Gap arising from Dataset-Specific Optimization and the long-tailed distribution of text categories. It proposes Joint-Dataset Learning (JDL) to replace single-dataset fine-tuning and introduces the Long-Tailed Benchmark (LTB) to rigorously evaluate detectors on 13 challenging tail categories, complemented by a self-supervised baseline MAEDet. Experiments across multiple detectors and datasets show that JDL improves cross-dataset generalization and reduces overfitting, while MAEDet enhances representation learning for long-tailed text, with substantial gains on LTB and reduced performance gaps between datasets. The work provides a practical, reproducible framework and code to advance robust scene text detection in diverse real-world conditions.

Abstract

Scene text detection has seen the emergence of high-performing methods that excel on academic benchmarks. However, these detectors often fail to replicate such success in real-world scenarios. We uncover two key factors contributing to this discrepancy through extensive experiments. First, a \textit{Fine-tuning Gap}, where models leverage \textit{Dataset-Specific Optimization} (DSO) paradigm for one domain at the cost of reduced effectiveness in others, leads to inflated performances on academic benchmarks. Second, the suboptimal performance in practical settings is primarily attributed to the long-tailed distribution of texts, where detectors struggle with rare and complex categories as artistic or overlapped text. Given that the DSO paradigm might undermine the generalization ability of models, we advocate for a \textit{Joint-Dataset Learning} (JDL) protocol to alleviate the Fine-tuning Gap. Additionally, an error analysis is conducted to identify three major categories and 13 subcategories of challenges in long-tailed scene text, upon which we propose a Long-Tailed Benchmark (LTB). LTB facilitates a comprehensive evaluation of ability to handle a diverse range of long-tailed challenges. We further introduce MAEDet, a self-supervised learning-based method, as a strong baseline for LTB. The code is available at https://github.com/pd162/LTB.

Figures (13)

• Figure 1: (a): The performance tendency of scene text detectors in recent years. We choose several representative methods in chronological order. Circles indicate the performances of other methods. (b): A visualization of fine-tuning gap. Each bar shows a performance comparison on the same benchmark. "$*$'’ denotes we fine-tuned the model with the open-source pretrained weight.
• Figure 2: A quantitative analysis of failure cases. The derivation of results is described in \ref{['sec:benchmark']}. The left figure shows that the overall detection results are divided into ignored, detected, and failed types. The middle presents statistics of failure cases with human annotation. The right illustrates the distribution of challenging text instances. The brightness of color indicates the prevalence of each category, with darker colors representing higher proportions. BG for background and Char for character.
• Figure 3: Comparison of the Dataset-Specific Optimization (DSO) and Joint-Dataset Learning (JDL) paradigms.
• Figure 4: An overview of LTB which collects the challenging issues in scene text detection, involving the Intra-Instance Problems, Inter-Instance Problems, and Background Problems. Except for the others category, there are 12 kinds of challenges in total, each is illustrated with two images. Left: a complete scene featuring challenging text instances, highlighted by red masks and arrows pointing to their locations. Right: a series of cropped challenging text instances, with the corresponding instance from the left image highlighted by a bold border.
• Figure 5: A detailed illustration of MAEDet architecture and training objectives. As indicated by its name, MAEDet is based on Mask Auto-Encoder (MAE), with the DBNet detection head adopted for text detection.