Interfacing Foundation Models' Embeddings

TL;DR

The paper tackles the limitation that foundation models are often specialized and difficult to connect across modalities, proposing FIND to align vision-language embeddings into an interleaved shared space for joint pixel- and image-level understanding. FIND uses a lightweight, trainable interface built on prompts and learnable queries, with content and conditional attention, projecting outputs into semantic and pixel spaces to support segmentation, grounding, and retrieval. A new FIND-Bench COCO-derived dataset provides interleaved annotations for training and evaluation, and empirical results show state-of-the-art performance on interleave retrieval and grounding, with competitive results on standard tasks. The work demonstrates practical applications such as interleave album search, video localization, and 3D feature-field construction, highlighting the potential for generalizable, extendable multimodal reasoning in real-world settings.

Abstract

Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in the teaser figure, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings.

Figures (7)

• Figure 1: The proposed FIND interface is generalizable to tasks that span granularity (pixel to image) and modality (vision to language). The retrieval space for this figure is the COCO validation set.
• Figure 2: (1) The concept of interfacing foundation models embedding, the black arrow means active attached modules and the gray arrow means the option that it can switch to. On the right, we show the difference of Multimodal and Interleave (2.a) in the context of embeddings matching; (2.b) in the context of embeddings interaction for reasoning and generation.
• Figure 3: Task Unification for retrieval, grounding, and segmentation. The corresponding components are labeled with the same color or connected with a line or arrow.
• Figure 4: (a) Preliminaries on the terminology of prompts and queries. (b) FIND approach pipeline. The shape of different polygons represents different embedding types, and the color (vision, language) of the polygons represents input modality. (c) Detailed architecture of the FIND Interface.
• Figure :