Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use

TL;DR

Modeling Collaborator addresses the challenge of training subjective vision concepts with minimal human labeling by integrating large language models and vision-language models to decompose concepts into objective components, automatically mine training data, and label via a VQA-guided workflow. The approach yields a distillation-based, lightweight classifier trained with active learning, outperforming Agile Modeling and zero-shot baselines, particularly on harder, subjective concepts. It significantly reduces manual effort (down to ~$100$ annotations) and eliminates crowd-sourced labeling in many cases, enabling rapid deployment in cost-sensitive applications. The results demonstrate strong performance on 15 subjective concepts across two public datasets, with notable gains in hard cases and robust human–machine alignment.

Abstract

From content moderation to wildlife conservation, the number of applications that require models to recognize nuanced or subjective visual concepts is growing. Traditionally, developing classifiers for such concepts requires substantial manual effort measured in hours, days, or even months to identify and annotate data needed for training. Even with recently proposed Agile Modeling techniques, which enable rapid bootstrapping of image classifiers, users are still required to spend 30 minutes or more of monotonous, repetitive data labeling just to train a single classifier. Drawing on Fiske's Cognitive Miser theory, we propose a new framework that alleviates manual effort by replacing human labeling with natural language interactions, reducing the total effort required to define a concept by an order of magnitude: from labeling 2,000 images to only 100 plus some natural language interactions. Our framework leverages recent advances in foundation models, both large language models and vision-language models, to carve out the concept space through conversation and by automatically labeling training data points. Most importantly, our framework eliminates the need for crowd-sourced annotations. Moreover, our framework ultimately produces lightweight classification models that are deployable in cost-sensitive scenarios. Across 15 subjective concepts and across 2 public image classification datasets, our trained models outperform traditional Agile Modeling as well as state-of-the-art zero-shot classification models like ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.

Figures (6)

• Figure 1: We introduce Modeling Collaborator: a framework that allows anyone to train vision models using natural language interactions and minimal effort. We show that today's best models (e.g. PaLI-X chen2023pali) change their answers depending on the prompt when classifying subjective concepts like gourmet tuna. Meanwhile, Modeling Collaborator uses LLMs and tool-use to train vision models by interacting with users to carve out the concept space.
• Figure 2: Modeling Collaborator Annotator system. For a given image, concept name, and description, the Annotator outputs a positive or negative label. Based on the name and description of the concept, the LLM generates relevant atomic questions to ask a VQA model (PaLI VQA in our case) (step A). These questions are fed into the VQA model that typically outputs a yes/no short answer (Step B). Additionally, we use a captioning version of PaLI (Step C) to generate a detailed description capturing as much detail as possible from the image. Finally, the LLM goes through a chain-of-thought reasoning process to output a decision and rationale (Step D).
• Figure 3: Modeling Collaborator Annotator examples for the concepts gourmet tuna (first row) and stop sign (second row). Hard negatives mined from the LAION dataset are shown in addition to some actual positives for the visual concept. The Modeling Collaborator Annotator is able to label the images as positive or negative as well as provide rationale. In some instances, the rationale could be incorrect (highlighted in red) due to error in VQA responses or hallucinations from the LLMs. Some of the reasons have been truncated for brevity.
• Figure 4: Comparing the contribution of increasingly more training examples annotated by crowd-annotators vs. Modeling Collaborator Annotator (fully automated). The y-axis shows the performance of the final distilled model. When user feedback is minimal (100 annotated examples), more crowd-annotators examples improve the final distilled model despite the noisy prediction. Modeling Collaborator Annotator provides similar improvement of performance without any human interactions and can be scaled better to annotate a lot more examples due to its autonomy.
• Figure 5: The impact of adding additional automatically annotated images on the final model quality (using the auPR metric). 100 user-annotated examples are used in addition to the thousands of Modeling Collaborator examples.