Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting

Xiaowen Zhang, Zijie Yue, Yong Luo, Cairong Zhao, Qijun Chen, Miaojing Shi

TL;DR

WS-COC is proposed, the first MLLM-driven weakly-supervised framework for class-agnostic object counting and shows that it matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs.

Abstract

Object counting is a fundamental task in computer vision, with broad applicability in many real-world scenarios. Fully-supervised counting methods require costly point-level annotations per object. Few weakly-supervised methods leverage only image-level object counts as supervision and achieve fairly promising results. They are, however, often limited to counting a single category, e.g. person. In this paper, we propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting. Instead of directly fine-tuning MLLMs to predict object counts, which can be challenging due to the modality gap, we incorporate three simple yet effective strategies to bootstrap the counting paradigm in both training and testing: First, a divide-and-discern dialogue tuning strategy is proposed to guide the MLLM to determine whether the object count falls within a specific range and progressively break down the range through multi-round dialogue. Second, a compare-and-rank count optimization strategy is introduced to train the MLLM to optimize the relative ranking of multiple images according to their object counts. Third, a global-and-local counting enhancement strategy aggregates and fuses local and global count predictions to improve counting performance in dense scenes. Extensive experiments on FSC-147, CARPK, PUCPR+, and ShanghaiTech show that WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs. Code is available at https://github.com/viscom-tongji/WS-COC.

Bootstrapping MLLM for Weakly-Supervised Class-Agnostic Object Counting

TL;DR

WS-COC is proposed, the first MLLM-driven weakly-supervised framework for class-agnostic object counting and shows that it matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs.

Abstract

Object counting is a fundamental task in computer vision, with broad applicability in many real-world scenarios. Fully-supervised counting methods require costly point-level annotations per object. Few weakly-supervised methods leverage only image-level object counts as supervision and achieve fairly promising results. They are, however, often limited to counting a single category, e.g. person. In this paper, we propose WS-COC, the first MLLM-driven weakly-supervised framework for class-agnostic object counting. Instead of directly fine-tuning MLLMs to predict object counts, which can be challenging due to the modality gap, we incorporate three simple yet effective strategies to bootstrap the counting paradigm in both training and testing: First, a divide-and-discern dialogue tuning strategy is proposed to guide the MLLM to determine whether the object count falls within a specific range and progressively break down the range through multi-round dialogue. Second, a compare-and-rank count optimization strategy is introduced to train the MLLM to optimize the relative ranking of multiple images according to their object counts. Third, a global-and-local counting enhancement strategy aggregates and fuses local and global count predictions to improve counting performance in dense scenes. Extensive experiments on FSC-147, CARPK, PUCPR+, and ShanghaiTech show that WS-COC matches or even surpasses many state-of-art fully-supervised methods while significantly reducing annotation costs. Code is available at https://github.com/viscom-tongji/WS-COC.
Paper Structure (18 sections, 1 equation, 9 figures, 11 tables)

This paper contains 18 sections, 1 equation, 9 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Object counting
  4. Multimodal Large Models
  5. Preliminary
  6. Method
  7. Divide-and-discern Dialogue Tuning
  8. Compare-and-rank Count Optimization
  9. Global-and-local counting enhancement
  10. Training and Inference
  11. Experiments
  12. Experiment details
  13. Comparison with state-of-the-art methods
  14. Ablation study
  15. Conclusion
  16. ...and 3 more sections

Figures (9)

  • Figure 1: (a)-(c) Visualizations results for MLLM-Zero, WS‑COC‑Base and WS‑COC. (d) MAE results of MLLM-Zero, WS‑COC‑Base, WS‑COC, and two state-of-art fully‑supervised counting methods (T2ICount qian2025t2icount and CountGD AminiNaieni24) on sparse (up to 20 instances per image) and dense (more than 100 instances per image) object scenes. WS‑COC significantly outperforms MLLM-Zero and WS‑COC‑Base, and yields competitive performance to T2ICount and CountGD. Experiment is on FSC-147 ranjan2021learning and MLLM is LLaVA-OneVersion-7B li2024llava.
  • Figure 2: Overview of the proposed WS‑COC for weakly‑supervised class‑agnostic object counting. (a) Divide‑and‑discern dialogue tuning (Sec. \ref{['bit']}) guides the MLLM to determine whether the object count falls within a specific range and progressively break down the range through multi-round dialogue. (b) Compare‑and‑rank count optimization (Sec. \ref{['rit']}) trains the MLLM to rank images by their object counts. (c) Global‑and‑local counting enhancement (Sec. \ref{['test time']}) aggregates and fuses the global and local count predictions to improve the performance in dense scenes during inference.
  • Figure 3: Visualization results among ground truth, CLIP-Count jiang2023clip, T2ICount qian2025t2icount, CountGD AminiNaieni24, and WS-COC.
  • Figure 4: MAE on the FSC-147 validation and test sets when varying (a) the ratio of $\delta$ in D$^3$T, (b) the $K$ in CRCO, and (c) the dense threshold $c^\text{h}$ in GLCE.
  • Figure 5: Attention map comparisons for MLLM-Zero, WS-COC-Base and WS-COC.
  • ...and 4 more figures