K-LITE: Learning Transferable Visual Models with External Knowledge

TL;DR

K-LITE presents a knowledge-augmented approach to learning transferable visual representations by incorporating WordNet and Wiktionary knowledge into language supervision used in pre-training and evaluation. The method leverages knowledge prompts and an adapter-based modular design to maintain consistency between training and inference, enabling improved zero-shot and few-shot transfer for image classification and object detection. Empirical results on 20 IC and 13 OD datasets demonstrate significant transfer gains and sample efficiency, with Wiktionary-based knowledge often providing the strongest improvements. The approach offers a scalable, knowledge-informed path to enhance vision-language models without requiring enormous increases in data collection.

Abstract

The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via large-scale data collection process. Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efficiency. We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts. In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is available at https://github.com/microsoft/klite.

Figures (7)

• Figure 1: Motivating examples: knowledge explains the content of the rare dish concepts.
• Figure 2: Left: Illustration of data construction process of the proposed knowledge-augmented language-image learning, in contrast to the baseline language-image learning. The query $q$ is constructed from Eq.\ref{['eq_query_construction']}. The same process is performed for both pre-training and downstream tasks. Right: The proposed strategy is applied to IC and OD for task-level transfer.
• Figure 3: Performance improvement analysis with external knowledge. External knowledge can largely improve concept overlap between pre-training and evaluation stages, hence usually yields higher recognition scores. Knowledge coverage indicates the percentage of concepts that exist in the knowledge base for each downstream dataset.
• Figure 4: Success and failure cases on image classification. For each image, the top row is the knowledge-based prediction, and the bottom row is the baseline prediction (no knowledge).
• Figure 5: The three datasets with the largest improvement according to Fig. \ref{['fig:22datasets']}: Flowers102, Food101 and OxfordPets. For each dataset, two success examples and one failure example are shown in (a) and (b) respectively. For each image, the top row is the knowledge-based prediction, and the bottom row is the baseline prediction (no knowledge).