CountGD++: Generalized Prompting for Open-World Counting

TL;DR

CountGD++ addresses open-world object counting by enabling negative prompts, automatic generation of visual exemplars (pseudo-exemplars), and adoption of external exemplars, all within a unified transformer-based counting model. The architecture integrates positive/negative text and visual prompts, external exemplar streams, and a cross-modality processing pipeline that yields 900 object queries and robust filtering. Training combines a multi-term loss with a contrastive component, mosaic data augmentation, and Hungarian matching to align predictions, while inference uses adaptive cropping to scale to dense scenes and pseudo-exemplar feedback to refine counts. The paper also demonstrates how CountGD++ can serve as a counting expert agent for LLMs, enabling synthetic/external exemplars and iterative prompting for images and videos, achieving state-of-the-art results across several benchmarks. The results show broad generalization, efficiency gains, and potential impact across medicine, materials science, and agriculture.

Abstract

The flexibility and accuracy of methods for automatically counting objects in images and videos are limited by the way the object can be specified. While existing methods allow users to describe the target object with text and visual examples, the visual examples must be manually annotated inside the image, and there is no way to specify what not to count. To address these gaps, we introduce novel capabilities that expand how the target object can be specified. Specifically, we extend the prompt to enable what not to count to be described with text and/or visual examples, introduce the concept of `pseudo-exemplars' that automate the annotation of visual examples at inference, and extend counting models to accept visual examples from both natural and synthetic external images. We also use our new counting model, CountGD++, as a vision expert agent for an LLM. Together, these contributions expand the prompt flexibility of multi-modal open-world counting and lead to significant improvements in accuracy, efficiency, and generalization across multiple datasets. Code is available at https://github.com/niki-amini-naieni/CountGDPlusPlus.

Figures (15)

• Figure 1: New capabilities of CountGD++.(a) Counting with Positive & Negative Prompts: The negative visual exemplar enables CountGD++ to differentiate between cells that have the same round shape as the object to count but are of a different appearance; (b) Pseudo-Exemplars: Pseudo-exemplars are automatically detected from text only and fed back to the model, improving the accuracy of the final count for objects, like unfamiliar fruits, that are challenging to identify given text alone.
• Figure 2: Inference with CountGD++. At inference, the object to be counted can be specified by a positive text prompt and any number of positive and negative visual exemplars and text prompts. The model outputs bounding boxes that are enumerated to estimate the count for objects matching the positive prompts. The input image and the image from which the exemplars are obtained (optionally the same as the input image) are passed through the image encoder, $f_{\boldsymbol{\theta_{SwinT}}}$, to obtain image tokens, spatial feature maps. The visual exemplar tokens are cropped out of the exemplar image feature map using RoIAlign in the Exemplar Extraction Module. The positive and negative texts are passed through the text encoder, $f_{\boldsymbol{\theta_{TT}}}$, to obtain text tokens. In the feature enhancer, $f_{\boldsymbol{\varphi}}$, the positive visual exemplar and positive text tokens are fused together with self-attention. Separately, the negative visual exemplar and negative text tokens are fused together with self-attention. The fused prompt features then cross-attend to the input image features. Further interaction occurs between the input image features and the prompt features in $f_{\boldsymbol{\psi}}$, which outputs enhanced prompt features and object queries, candidate instances that map to object boxes for all the objects specified by both the positive and negative prompts. The Object Filtering Module removes object queries that score below a confidence threshold or are more similar to negative prompts than positive prompts. The remaining object queries are enumerated to estimate the final count. The architecture is built on that of Grounding DINO liu2023grounding.
• Figure 3: (a) Examples of positive and negative prompts. Any combination is valid. (b) Self-attention between prompt features. In the Feature Enhancer, corresponding visual exemplar and text features self-attend to each other but not to other visual exemplar and text features. Negative prompts do not attend to each other if they describe different classes.
• Figure 4: (a) Pipeline for generating and counting with synthetic exemplars using an LLM and CountGD++. (b) Example of iteratively improving the count with pseudo-exemplars.
• Figure 5: Counting growing and deforming crystals in x-ray videos with pseudo-exemplars. CountGD++ is applied to each frame. In the initial frame, only text is provided. For subsequent frames, the top 3 highest scoring boxes from the previous frame are selected as pseudo-exemplars and input to CountGD++ together with text to predict boxes for the current frame.