Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Knowledge Graphs to harvest datasets for efficient CLIP model training

Simon Ging, Sebastian Walter, Jelena Bratulić, Johannes Dienert, Hannah Bast, Thomas Brox

TL;DR

This work tackles the data-hungry nature of CLIP by presenting a knowledge-graph–guided dataset harvesting pipeline that enables effective pretraining with far less data. By linking Wikidata and WordNet to an LLM-driven search process, the authors create EntityNet, a 33M-image, 45M-alt-text vision-language dataset with a 10M-organism subset, enabling from-scratch CLIP training on modest hardware in under a day. Across generic and expert domains, EntityNet-based CLIP models achieve state-of-the-art or competitive results with substantially reduced compute compared to conventional large-scale datasets, and finetuning on expert data can further boost performance. The approach demonstrates that structured knowledge graphs can automate high-quality data assembly for domain-specific CLIP models, offering a scalable path to controlled, efficient vision-language pretraining and broader applicability beyond natural images.

Abstract

Training high-quality CLIP models typically requires enormous datasets, which limits the development of domain-specific models -- especially in areas that even the largest CLIP models do not cover well -- and drives up training costs. This poses challenges for scientific research that needs fine-grained control over the training procedure of CLIP models. In this work, we show that by employing smart web search strategies enhanced with knowledge graphs, a robust CLIP model can be trained from scratch with considerably less data. Specifically, we demonstrate that an expert foundation model for living organisms can be built using just 10M images. Moreover, we introduce EntityNet, a dataset comprising 33M images paired with 46M text descriptions, which enables the training of a generic CLIP model in significantly reduced time.

Using Knowledge Graphs to harvest datasets for efficient CLIP model training

TL;DR

This work tackles the data-hungry nature of CLIP by presenting a knowledge-graph–guided dataset harvesting pipeline that enables effective pretraining with far less data. By linking Wikidata and WordNet to an LLM-driven search process, the authors create EntityNet, a 33M-image, 45M-alt-text vision-language dataset with a 10M-organism subset, enabling from-scratch CLIP training on modest hardware in under a day. Across generic and expert domains, EntityNet-based CLIP models achieve state-of-the-art or competitive results with substantially reduced compute compared to conventional large-scale datasets, and finetuning on expert data can further boost performance. The approach demonstrates that structured knowledge graphs can automate high-quality data assembly for domain-specific CLIP models, offering a scalable path to controlled, efficient vision-language pretraining and broader applicability beyond natural images.

Abstract

Training high-quality CLIP models typically requires enormous datasets, which limits the development of domain-specific models -- especially in areas that even the largest CLIP models do not cover well -- and drives up training costs. This poses challenges for scientific research that needs fine-grained control over the training procedure of CLIP models. In this work, we show that by employing smart web search strategies enhanced with knowledge graphs, a robust CLIP model can be trained from scratch with considerably less data. Specifically, we demonstrate that an expert foundation model for living organisms can be built using just 10M images. Moreover, we introduce EntityNet, a dataset comprising 33M images paired with 46M text descriptions, which enables the training of a generic CLIP model in significantly reduced time.
Paper Structure (24 sections, 5 figures, 14 tables)

This paper contains 24 sections, 5 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Entity extraction
  4. Attribute generation
  5. Query building
  6. Image search and filtering
  7. Experimental setup
  8. Training
  9. Evaluated models
  10. Object classification evaluation
  11. Retrieval evaluation
  12. Results
  13. Limitations
  14. Conclusions
  15. Additional analyses
  16. ...and 9 more sections

Figures (5)

  • Figure 1: We demonstrate how to harvest datasets for training CLIP models with an improved quality-cost trade-off, for a generic (left) or an expert domain (right).
  • Figure 2: We create a dataset for vision-language pretraining: First, we extract entities from knowledge graphs, then generate attributes and natural types for them. We search for different combinations of entities, attributes, and types in image search engines, and collect alt texts for each image. Finally, we train our model on the combined data.
  • Figure 3: Results on image retrieval and distribution shift robustness on ImageNet.
  • Figure 4: Scaling entities and queries per entity.
  • Figure 5: Generic SPARQL query for extracting entities from Wikidata that are related to a given set of super-entities. The super-entities are manually set within the VALUES ?typ { ... } clause. In this example it is the motor car entity wd:Q42889. A minimum number of sitelinks can also be specified to filter out unpopular entities, here it is set to 5.