Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-shot target-driven instance detection based on open-vocabulary object detection models

Ben Crulis, Barthelemy Serres, Cyril De Runz, Gilles Venturini

TL;DR

This work proposes a lightweight method to turn the YOLO-World model into a one-shot or few-shot object recognition models without requiring textual descriptions, and shows that performance increases with the model size, the number of examples and the use of image augmentation.

Abstract

Current large open vision models could be useful for one and few-shot object recognition. Nevertheless, gradient-based re-training solutions are costly. On the other hand, open-vocabulary object detection models bring closer visual and textual concepts in the same latent space, allowing zero-shot detection via prompting at small computational cost. We propose a lightweight method to turn the latter into a one-shot or few-shot object recognition models without requiring textual descriptions. Our experiments on the TEgO dataset using the YOLO-World model as a base show that performance increases with the model size, the number of examples and the use of image augmentation.

Few-shot target-driven instance detection based on open-vocabulary object detection models

TL;DR

This work proposes a lightweight method to turn the YOLO-World model into a one-shot or few-shot object recognition models without requiring textual descriptions, and shows that performance increases with the model size, the number of examples and the use of image augmentation.

Abstract

Current large open vision models could be useful for one and few-shot object recognition. Nevertheless, gradient-based re-training solutions are costly. On the other hand, open-vocabulary object detection models bring closer visual and textual concepts in the same latent space, allowing zero-shot detection via prompting at small computational cost. We propose a lightweight method to turn the latter into a one-shot or few-shot object recognition models without requiring textual descriptions. Our experiments on the TEgO dataset using the YOLO-World model as a base show that performance increases with the model size, the number of examples and the use of image augmentation.

Paper Structure

This paper contains 20 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. YOLO-World
  4. Other Object Detection Models Supporting Prompting
  5. Transforming YOLO-World into a Target Driven Instance Detector
  6. Model Conversion
  7. Inference
  8. Experiments
  9. Object recognition on TEgO
  10. Protocol
  11. Results
  12. Analysis
  13. Improving the Results using Whitening and Coloring
  14. Whitening and Coloring.
  15. Results.
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overview of the different detection paradigms. Our proposed method falls into the TDID paradigm (c). The traditional object detection paradigm (a) produces models with a fixed set of object classes that can be recognized, which gives no control over the objects that can be detected or not to the user. The Open Vocabulary paradigm (b) gives some control to the user by allowing them to use natural language text prompts that specify the classes the model should detect. Finally, the Target Driven Instance Detection paradigm (c) proposes to directly use one or more image examples of the target object as prompt for the model so that it can detect the object in other images.
  • Figure 2: Overview of the proposed method. In the training phase (a), YOLO-World is used with a text prompt such as "main object" to detect the most salient object in the image and use the the inferred bounding box to crop the image to the object. The cropped image is then optionally augmented and the resulting embeddings are aggregated to get a single vector per saved object. In the inference phase (b), the saved object embeddings are used as prompt in YOLO-World, which allows detecting the saved objects in other images.
  • Figure 3: UMAP plot of the CLIP embeddings computed from the cropped TEgO images.
  • Figure 4: Confusion matrix on the test set
  • Figure 5: Histograms of the predicted probabilities for incorrect classifications