Compositional Caching for Training-free Open-vocabulary Attribute Detection

TL;DR

ComCa addresses open-vocabulary attribute detection without supervision by building a compositional image cache that pairs attributes with objects using web-scale data and LLM reasoning. A Vision-Language Model (e.g., CLIP) is refined at inference time by aggregating soft attribute labels from the cache and by fusing cache-derived scores with standard CLIP predictions, all without training data. The core ideas are to estimate attribute–object compatibility to populate a compact, per-attribute cache, to soft-label cache entries to reflect real-world co-occurrence of attributes, and to demonstrate model-agnostic gains across OVAD and VAW with multiple backbones. Experiments show that ComCa significantly surpasses zero-shot and cache-based baselines and competes with training-based methods, while exhibiting robustness to dataset biases and architectural choices. The final score combines cache-based predictions and CLIP via $f(x,a)=\eta_A\left(\lambda f^{\textsc{ComCa}}_\mathcal{C}(x,a) + f^{\text{CLIP}}_\mathcal{C}(x,a)\right)$, enabling scalable open-vocabulary attribute detection in practice.

Abstract

Attribute detection is crucial for many computer vision tasks, as it enables systems to describe properties such as color, texture, and material. Current approaches often rely on labor-intensive annotation processes which are inherently limited: objects can be described at an arbitrary level of detail (e.g., color vs. color shades), leading to ambiguities when the annotators are not instructed carefully. Furthermore, they operate within a predefined set of attributes, reducing scalability and adaptability to unforeseen downstream applications. We present Compositional Caching (ComCa), a training-free method for open-vocabulary attribute detection that overcomes these constraints. ComCa requires only the list of target attributes and objects as input, using them to populate an auxiliary cache of images by leveraging web-scale databases and Large Language Models to determine attribute-object compatibility. To account for the compositional nature of attributes, cache images receive soft attribute labels. Those are aggregated at inference time based on the similarity between the input and cache images, refining the predictions of underlying Vision-Language Models (VLMs). Importantly, our approach is model-agnostic, compatible with various VLMs. Experiments on public datasets demonstrate that ComCa significantly outperforms zero-shot and cache-based baselines, competing with recent training-based methods, proving that a carefully designed training-free approach can successfully address open-vocabulary attribute detection.

Figures (9)

• Figure 1: Example of attribute annotations. Attribute annotations are often: sparse, as they are not consistent across samples; incomplete, as not all attributes are annotated; ambiguous, as they can be subjective or miss a frame of reference. This makes open-vocabulary attribute detection challenging. Differently from previous works bravo2023ovadchen2023ovarnet, we do not rely on such annotations and propose ComCa, a training-free approach requiring no supervision.
• Figure 2: ComCa's cache construction. Given a list of attributes and objects, we compute their compatibility from a large database $D_r$ and with an LLM. The scores are merged and normalized to obtain the compatibility distribution, from which we sample cache entries and construct the cache. We enrich the latter with soft labels from the VLM-based similarity between cache images and attributes.
• Figure 3: Qualitative results. Predictions of OVAD, CLIP and ComCa on some OVAD images. Green are correct ones, red are wrong.
• Figure 4: Cross-dataset results. Results on OVAD (left) and VAW (right). Yellow indicates methods tested cross-domain, i.e., train and test set differ. Red indicates methods tested in-domain, and green our training-free method, which does not suffer from domain shift problems. All experiments use a RN50 backbone.
• Figure 4: Ablation on cache construction. Green indicates our default configuration. Bold indicates the best results. Results are without soft labels. See the Supp. Mat. for an extended version of the table.