TSAL: Few-shot Text Segmentation Based on Attribute Learning

TL;DR

TSAL introduces a two-branch framework that leverages CLIP priors for few-shot scene text segmentation. The visual-guided branch builds a robust visual feature bank, while the adaptive prompt-guided branch learns text attributes through learnable prompts and LLM-generated descriptors, all aligned by Adaptive Feature Alignment. Inference fuses branch-specific scores with minimal supervision, yielding state-of-the-art performance on TextSeg, Total-Text, and ICDAR13 FST under 1/2/4-shot settings. The approach is simple, data-efficient, and demonstrates strong generalization to diverse text styles and backgrounds, with potential applications across text-related domains.

Abstract

Recently supervised learning rapidly develops in scene text segmentation. However, the lack of high-quality datasets and the high cost of pixel annotation greatly limit the development of them. Considering the well-performed few-shot learning methods for downstream tasks, we investigate the application of the few-shot learning method to scene text segmentation. We propose TSAL, which leverages CLIP's prior knowledge to learn text attributes for segmentation. To fully utilize the semantic and texture information in the image, a visual-guided branch is proposed to separately extract text and background features. To reduce data dependency and improve text detection accuracy, the adaptive prompt-guided branch employs effective adaptive prompt templates to capture various text attributes. To enable adaptive prompts capture distinctive text features and complex background distribution, we propose Adaptive Feature Alignment module(AFA). By aligning learnable tokens of different attributes with visual features and prompt prototypes, AFA enables adaptive prompts to capture both general and distinctive attribute information. TSAL can capture the unique attributes of text and achieve precise segmentation using only few images. Experiments demonstrate that our method achieves SOTA performance on multiple text segmentation datasets under few-shot settings and show great potential in text-related domains.

Figures (6)

• Figure 1: Training and Inference Pipeline of TSAL under 1-shot setting. During training, we optimize the foreground and background prompts to learn attribute features. During inference, we calculate the similarity between the optimized features and the input image, obtaining the final result.
• Figure 2: Overview of TSAL (2-shot): the Visual-guided branch extracts and stores foreground and background features in visual feature bank for inference. The adaptive prompt-guided branch design effective templates for adaptive prompts and utilizes LLMs to generate fine-grained descriptions for each input image. AFA aligns the adaptive prompts with visual features and prompt prototypes, enabling them to capture rich semantic information. The optimized prompt features are stored in the prompt feature bank for inference.
• Figure 3: Visualization results under the 1-shot setting on the TextSeg, Total-Text, and IDAR13 FST datasets. To guarantee fairness, PromptAD uses our mask generator to generate the masks. Our method performs well in a variety of complex scenes.
• Figure 4: Visualization examples of AFA for four different scenarios. The first row is w/o AFA, the second row is w/ AFA.
• Figure 5: FgIoU of TextSeg in 1-shot setting using different numbers of prompts and ratios of prototypes to learnable prompts.