Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DetailCLIP: Injecting Image Details into CLIP's Feature Space

Zilun Zhang, Cuifeng Shen, Yuan Shen, Huixin Xiong, Xinyu Zhou, Tiancheng Zhao, Jianwei Yin

TL;DR

DetailCLIP addresses the loss of fine image details in CLIP-like models caused by fixed input resolutions. It introduces a patch-based Complete Cover strategy and a transformer-based fusion to produce a single, detail-rich feature that remains in CLIP's semantic space, trained with a proxy loss derived from class-promoted text prompts. The approach yields substantial improvements in detail-driven text-to-image retrieval on CLEVR-DS and real-world datasets, demonstrating stronger grounding of small or cluttered objects. This framework offers practical benefits for fine-grained retrieval and opens avenues for end-to-end patch-aware representations in large-scale vision-language systems.

Abstract

Although CLIP-like Visual Language Models provide a functional joint feature space for image and text, due to the limitation of the CILP-like model's image input size (e.g., 224), subtle details are lost in the feature representation if we input high-resolution images (e.g., 2240). In this work, we introduce an efficient framework that can produce a single feature representation for a high-resolution image that injects image details and shares the same semantic space as the original CLIP. In the framework, we train a feature fusing model based on CLIP features extracted from a carefully designed image patch method that can cover objects of any scale, weakly supervised by image-agnostic class prompted queries. We validate our framework by retrieving images from class prompted queries on the real world and synthetic datasets, showing significant performance improvement on these tasks. Furthermore, to fully demonstrate our framework's detail retrieval ability, we construct a CLEVR-like synthetic dataset called CLVER-DS, which is fully annotated and has a controllable object scale.

DetailCLIP: Injecting Image Details into CLIP's Feature Space

TL;DR

DetailCLIP addresses the loss of fine image details in CLIP-like models caused by fixed input resolutions. It introduces a patch-based Complete Cover strategy and a transformer-based fusion to produce a single, detail-rich feature that remains in CLIP's semantic space, trained with a proxy loss derived from class-promoted text prompts. The approach yields substantial improvements in detail-driven text-to-image retrieval on CLEVR-DS and real-world datasets, demonstrating stronger grounding of small or cluttered objects. This framework offers practical benefits for fine-grained retrieval and opens avenues for end-to-end patch-aware representations in large-scale vision-language systems.

Abstract

Although CLIP-like Visual Language Models provide a functional joint feature space for image and text, due to the limitation of the CILP-like model's image input size (e.g., 224), subtle details are lost in the feature representation if we input high-resolution images (e.g., 2240). In this work, we introduce an efficient framework that can produce a single feature representation for a high-resolution image that injects image details and shares the same semantic space as the original CLIP. In the framework, we train a feature fusing model based on CLIP features extracted from a carefully designed image patch method that can cover objects of any scale, weakly supervised by image-agnostic class prompted queries. We validate our framework by retrieving images from class prompted queries on the real world and synthetic datasets, showing significant performance improvement on these tasks. Furthermore, to fully demonstrate our framework's detail retrieval ability, we construct a CLEVR-like synthetic dataset called CLVER-DS, which is fully annotated and has a controllable object scale.
Paper Structure (26 sections, 8 equations, 3 figures, 5 tables)

This paper contains 26 sections, 8 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language model overview
  4. CLIP-like models
  5. Vision Language Model applications
  6. Methodology
  7. Motivation and Effective Scale Sensitivity
  8. Problem Definition
  9. Complete Cover
  10. Model & Loss
  11. Fusing Model
  12. Query Proxy Loss
  13. Benchmark
  14. Existing Datasets
  15. CLEVR-DS
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Retrieval results from CLIP model and our DetailCLIP model. DetailCLIP retrieves more images with small target objects.
  • Figure 2: Illustration of Patch Selection of Complete Cover.Minimum effective object means the minimum object that can be retrieved by CLIP from the patch. Patches with different shapes will slide the whole image to cover objects equal to or bigger than the minimum effective size.
  • Figure 3: Multi Dataset Illustration