YOLO-RD: Introducing Relevant and Compact Explicit Knowledge to YOLO by Retriever-Dictionary

TL;DR

This work addresses the underutilization of dataset-wide information in real-time vision models by introducing the Retriever-Dictionary (RD) module, a lightweight plug-in that augments YOLO-based detectors with explicit knowledge from pre-built dictionaries. RD comprises a Coefficient Generator and a Global Information Exchanger that produce per-pixel coefficients to weight a dictionary of atoms, with outputs blended with input residuals and normalized via PN and WN operations, enabling efficient knowledge integration. Dictionary initialization leverages encoders from Vision Models, Vision-Language Models (CLIP), and Large Language Models (GPT prompts), followed by k-means selection and a SimCLR-based compression to keep the dictionary compact. Empirically, RD yields consistent mAP gains across 1-stage and 2-stage detectors, including Faster R-CNN and Deformable DETR, while also improving segmentation and classification, all with minimal parameter overhead and without requiring external loss terms. The results suggest RD’s potential to improve real-time vision systems by leveraging cross-modal and cross-domain knowledge in a scalable, efficient manner, with broad applicability to multiple architectures and tasks.

Abstract

Identifying and localizing objects within images is a fundamental challenge, and numerous efforts have been made to enhance model accuracy by experimenting with diverse architectures and refining training strategies. Nevertheless, a prevalent limitation in existing models is overemphasizing the current input while ignoring the information from the entire dataset. We introduce an innovative Retriever-Dictionary (RD) module to address this issue. This architecture enables YOLO-based models to efficiently retrieve features from a Dictionary that contains the insight of the dataset, which is built by the knowledge from Visual Models (VM), Large Language Models (LLM), or Visual Language Models (VLM). The flexible RD enables the model to incorporate such explicit knowledge that enhances the ability to benefit multiple tasks, specifically, segmentation, detection, and classification, from pixel to image level. The experiments show that using the RD significantly improves model performance, achieving more than a 3\% increase in mean Average Precision for object detection with less than a 1% increase in model parameters. Beyond 1-stage object detection models, the RD module improves the effectiveness of 2-stage models and DETR-based architectures, such as Faster R-CNN and Deformable DETR. Code is released at https://github.com/henrytsui000/YOLO.

Figures (10)

• Figure 1: A comparison between traditional models and our proposed RetrieverDictionary module. On the left, CNN-based models focus on local regions (blue box), while Transformer-based models tend to utilize the entire image (red box). However, both methods fail to leverage the information from the entire dataset, as illustrated in the bottom-right corner. Our module enhances the model's resource utilization by incorporating knowledge from other parts of the dataset.
• Figure 2: Dictionary is initialized by encoding the dataset using an image encoder, from which $N$ embeddings are selected as Dictionary atoms. During training, for each input feature $X_{w,h}$, the Retriever core—comprising the Coefficient Generator ($\mathbf{G}$) and the Global Information Exchanger ($\mathbf{E}$)—generates coefficients for each atom $\alpha$ in the Dictionary$D$. Then, the normalized coefficients are used as weights for each Dictionary atom. Finally, by concatenating the residual of $X_{w,h}$, the output $Y_{w,h}$ is obtained.
• Figure 3: Illustrates the distribution of the dataset in the model's middle layer, where the blue square represents the current input feature. In traditional models, only the input feature is used, neglecting the rich information available in the dataset. In contrast, with our Retriever-Dictionary model, additional data information is retrieved from the dataset. The dictionary can be initialized from different models: vision models, language models, or vision-language models. The latter provides a more comprehensive and integrated representation of the dataset.
• Figure 4: Visualization of Dictionary Atom Coefficients
• Figure 11: Visualization of w/o RD module