Expanding Zero-Shot Object Counting with Rich Prompts

TL;DR

RichCount tackles zero-shot object counting by addressing the misalignment between text prompts and visual features and the shallow semantic content of category labels. It introduces a two-stage training framework: Visual-Text Alignment, which enriches and aligns text with image features using an MLLM-derived descriptions, an FFN on the image encoder, and an adapter on the text encoder; and Text-Based Counting, which uses a fusion mechanism and a decoder to produce density maps from diverse prompts. The approach achieves state-of-the-art zero-shot performance on FSC-147 and demonstrates strong cross-dataset generalization to CARPK and ShanghaiTech, with ablations highlighting the critical roles of enriched descriptions, feature alignment, and consistent cross-prompt density estimation. By enabling flexible inference with category names, detailed descriptions, or attribute prompts, RichCount advances open-world counting and sets a solid foundation for robust visual-text counting in unseen categories.

Abstract

Expanding pre-trained zero-shot counting models to handle unseen categories requires more than simply adding new prompts, as this approach does not achieve the necessary alignment between text and visual features for accurate counting. We introduce RichCount, the first framework to address these limitations, employing a two-stage training strategy that enhances text encoding and strengthens the model's association with objects in images. RichCount improves zero-shot counting for unseen categories through two key objectives: (1) enriching text features with a feed-forward network and adapter trained on text-image similarity, thereby creating robust, aligned representations; and (2) applying this refined encoder to counting tasks, enabling effective generalization across diverse prompts and complex images. In this manner, RichCount goes beyond simple prompt expansion to establish meaningful feature alignment that supports accurate counting across novel categories. Extensive experiments on three benchmark datasets demonstrate the effectiveness of RichCount, achieving state-of-the-art performance in zero-shot counting and significantly enhancing generalization to unseen categories in open-world scenarios.

Figures (11)

• Figure 1: Illustration of Text-Visual Association. (a) Text-based counting methods (ClipCount jiang2023clip) often result in non-specific category estimations, whereas visual prompts (T-Rex jiang2023t) mitigate this issue. A natural modality and semantic gap exists between text and visual prompts. (b) Our method addresses this misalignment, enabling the use of diverse text prompts as inputs.
• Figure 2: Overview of the Proposed Method. The framework consists of two training stages: (1) Visual-Text Alignment, which utilizes ChatGPT to generate descriptive text for image categories. To align features, an FFN is added to the CLIP visual encoder, and an adapter is integrated into the CLIP text encoder; (2) Text-Based Counting, which freezes the encoders and trains the interaction module and decoder to ensure consistency between density maps generated from text descriptions and their corresponding textual inputs. During inference, the model generates density maps based on diverse textual prompts.
• Figure 3: Illustration of the Alignment Strategy. An adapter refines text embeddings, and an FFN processes visual features aligned via contrastive loss for cross-modal understanding.
• Figure 4: Illustration of Descriptions Generated by ChatGPT-4.
• Figure 5: Visualization of t-SNE Clusters Before and After Alignment. Misaligned clusters are circled in yellow.