PhotoScout: Synthesis-Powered Multi-Modal Image Search

TL;DR

PhotoScout tackles the problem of semantic, structured image retrieval in large photo collections by combining natural language input, positive/negative example images, and interactive object tagging. It grounds user intent through a neuro-symbolic domain-specific language and a synthesis pipeline that first generates program sketches with an LLM, then completes them via user grounding and enumerative search, finally executing the complete query on a dataset. The system enables fast, interactive refinement with grounding steps and provides natural-language explanations of results. A user study with 25 participants shows PhotoScout improves task accuracy and reduces manual effort compared with a CLIP-based baseline, highlighting the value of multi-modal specification for complex image-search tasks and suggesting directions for future fusion of open-ended and constraint-based search paradigms.

Abstract

Due to the availability of increasingly large amounts of visual data, there is a growing need for tools that can help users find relevant images. While existing tools can perform image retrieval based on similarity or metadata, they fall short in scenarios that necessitate semantic reasoning about the content of the image. This paper explores a new multi-modal image search approach that allows users to conveniently specify and perform semantic image search tasks. With our tool, PhotoScout, the user interactively provides natural language descriptions, positive and negative examples, and object tags to specify their search tasks. Under the hood, PhotoScout is powered by a program synthesis engine that generates visual queries in a domain-specific language and executes the synthesized program to retrieve the desired images. In a study with 25 participants, we observed that PhotoScout allows users to perform image retrieval tasks more accurately and with less manual effort.

Figures (10)

• Figure 1: Left: Three images that matches John's intent: the bride and groom are next to each other, with the bride holding flowers. Right: an image that is incorrect image because the bride is not holding flowers.
• Figure 2: The PhotoScout interface has six main panels: (1) The user enters a natural language query describing the images to be searched. (2) The example images panel highlights all the positive and negative images that the user has already labeled. Positive examples are wrapped in a green box and negative examples are wrapped in a red box. (3) The album preview panel displays all the photos in the album to be searched from. (4) Once the user selects a photo to label, the example labeling panel displays the image and the example labeling buttons. (5) The search results panel shows all the images that PhotoScout finds that match both the natural language description and the labeled examples, along with a natural language explanation. (6) The photo export panel shows all the images selected by the user as the final search results.
• Figure 3: The example labeling panel consists of 4 elements. (a) A view of the photo to be labeled. When a user hovers over it, each object identified by the detector is highlighted with a square box, with the detailed description of the detected object shown in (c). (b) asks the user to label the photo either as a positive or negative example. (d) is a tagging interface so the user can give semantic meanings to the detected face or object. In this particular example, the user is tagging the bride with the name "Alice" so that they can refer to the bride in the query.
• Figure 4: An example image.
• Figure 5: Image Search DSL. All predicates are binary except for HasRelation (ternary).

Theorems & Definitions (5)

• Example 4.1
• Example 4.2
• Example 4.3
• Example 4.4
• Example 4.5