Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions

Prajwal Gatti, Kshitij Parikh, Dhriti Prasanna Paul, Manish Gupta, Anand Mishra

TL;DR

CSTBIR addresses retrieving natural-scene images using composite queries that pair rough sketches of difficult-to-name objects with partial descriptive text. The authors propose STNet, a three-branch multimodal transformer built on CLIP that jointly encodes text, sketches, and images, guided by sketch-based localization and trained with five objectives including $\mathcal{L}_{CT}$, $\mathcal{L}_{CLS}$, $\mathcal{L}_{OD}$, and $\mathcal{L}_{SR}$. They introduce CSTBIR, a large dataset with ≈$2\times 10^{6}$ queries and ≈$1.08\times 10^{5}$ images, and demonstrate that STNet outperforms state-of-the-art TBIR, SBIR, and existing CSTBIR baselines across Test-1K, Test-5K, and Open-Category settings. The work provides a practical framework for composite multimodal queries and delivers strong generalization to unseen objects, with code and data released for replication and extension.

Abstract

Non-native speakers with limited vocabulary often struggle to name specific objects despite being able to visualize them, e.g., people outside Australia searching for numbats. Further, users may want to search for such elusive objects with difficult-to-sketch interactions, e.g., numbat digging in the ground. In such common but complex situations, users desire a search interface that accepts composite multimodal queries comprising hand-drawn sketches of difficult-to-name but easy-to-draw objects and text describing difficult-to-sketch but easy-to-verbalize object attributes or interaction with the scene. This novel problem statement distinctly differs from the previously well-researched TBIR (text-based image retrieval) and SBIR (sketch-based image retrieval) problems. To study this under-explored task, we curate a dataset, CSTBIR (Composite Sketch+Text Based Image Retrieval), consisting of approx. 2M queries and 108K natural scene images. Further, as a solution to this problem, we propose a pretrained multimodal transformer-based baseline, STNET (Sketch+Text Network), that uses a hand-drawn sketch to localize relevant objects in the natural scene image, and encodes the text and image to perform image retrieval. In addition to contrastive learning, we propose multiple training objectives that improve the performance of our model. Extensive experiments show that our proposed method outperforms several state-of-the-art retrieval methods for text-only, sketch-only, and composite query modalities. We make the dataset and code available at our project website.

Composite Sketch+Text Queries for Retrieving Objects with Elusive Names and Complex Interactions

TL;DR

CSTBIR addresses retrieving natural-scene images using composite queries that pair rough sketches of difficult-to-name objects with partial descriptive text. The authors propose STNet, a three-branch multimodal transformer built on CLIP that jointly encodes text, sketches, and images, guided by sketch-based localization and trained with five objectives including , , , and . They introduce CSTBIR, a large dataset with ≈ queries and ≈ images, and demonstrate that STNet outperforms state-of-the-art TBIR, SBIR, and existing CSTBIR baselines across Test-1K, Test-5K, and Open-Category settings. The work provides a practical framework for composite multimodal queries and delivers strong generalization to unseen objects, with code and data released for replication and extension.

Abstract

Non-native speakers with limited vocabulary often struggle to name specific objects despite being able to visualize them, e.g., people outside Australia searching for numbats. Further, users may want to search for such elusive objects with difficult-to-sketch interactions, e.g., numbat digging in the ground. In such common but complex situations, users desire a search interface that accepts composite multimodal queries comprising hand-drawn sketches of difficult-to-name but easy-to-draw objects and text describing difficult-to-sketch but easy-to-verbalize object attributes or interaction with the scene. This novel problem statement distinctly differs from the previously well-researched TBIR (text-based image retrieval) and SBIR (sketch-based image retrieval) problems. To study this under-explored task, we curate a dataset, CSTBIR (Composite Sketch+Text Based Image Retrieval), consisting of approx. 2M queries and 108K natural scene images. Further, as a solution to this problem, we propose a pretrained multimodal transformer-based baseline, STNET (Sketch+Text Network), that uses a hand-drawn sketch to localize relevant objects in the natural scene image, and encodes the text and image to perform image retrieval. In addition to contrastive learning, we propose multiple training objectives that improve the performance of our model. Extensive experiments show that our proposed method outperforms several state-of-the-art retrieval methods for text-only, sketch-only, and composite query modalities. We make the dataset and code available at our project website.

Paper Structure

This paper contains 22 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sketch-Based Image Retrieval (SBIR):
  4. Text-Based Image Retrieval (TBIR):
  5. Multimodal Query Based Image Retrieval:
  6. The CSTBIR Problem and Dataset
  7. The Proposed STNet Model for CSTBIR
  8. Query (Sketch+Text) Encoding
  9. Image Encoding
  10. STNet Training
  11. Contrastive Training ($\mathcal{L}_{CT}$)
  12. Object Classification ($\mathcal{L}_{CLS}^T$ and $\mathcal{L}_{CLS}^I$)
  13. Sketch-Guided Object Detection ($\mathcal{L}_{OD}$)
  14. Sketch Reconstruction ($\mathcal{L}_{SR}$)
  15. Experiments and Results
  16. ...and 7 more sections

Figures (4)

  • Figure 1: CSTBIR: Composite Sketch+Text Based Image Retrieval: A user wants to search "Numbat digging in the ground" but does not know the word "numbat", and the interaction "digging in the ground" is not easy to sketch.
  • Figure 2: Examples from our dataset -- CSTBIR. It contains queries composed of a sketch of an object, a natural language text describing its attributes and interactions, and the target natural scene image containing the object. Queried objects: markhor (left), and bodhran (right).
  • Figure 3: Overview of the proposed method, STNet for the CSTBIR problem.
  • Figure 4: Qualitative results of our STNet. We show top-5 retrieved results for the multimodal (sketch+text) queries shown in left most column. From top to bottom, the sketch are for capybara, sitar, penny-farthing, and okapi. The ground truth image is shown with a green frame. (Best viewed in color).