Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EBind: a practical approach to space binding

Jim Broadbent, Felix Cohen, Frederik Hvilshøj, Eric Landau, Eren Sasoglu

TL;DR

EBind tackles the problem of binding embedding spaces across text, image, video, audio, and point clouds with a resource-efficient, data-centric approach. It uses a simple architecture with frozen encoders per modality and small MLP projections, trained through a three-tier data strategy that combines auto-paired quintuplets, human-verified triples, and open-captioned data. With a compact 1.8B-parameter model, EBind achieves state-of-the-art performance on 13 benchmarks and introduces EShot, a high-quality zero-shot PC–audio evaluation benchmark, all while enabling single-GPU training in hours and releasing code and data openly. The work emphasizes data quality and practical training efficiency, offering a scalable path toward accessible, multi-modal binding and broad applicability in retrieval and cross-modal understanding.

Abstract

We simplify space binding by focusing on two core components, a single encoder per modality and high-quality data; enabling training state-of-the-art models on a single GPU in a few hours as opposed to multiple days. We present EBind, an Easy, data-centric, and parameter-efficient method to Bind the embedding spaces of multiple contrastive models. We demonstrate that a simple 1.8B-parameter image-text-video-audio-3D model can outperform models 4 to 17x the size. The key to achieving this is a carefully curated dataset of three complementary data sources: i) 6.7M fully-automated multimodal quintuples sourced via SOTA retrieval models, ii) 1M diverse, semi-automated triples annotated by humans as negative, partial, or positive matches, and iii) 3.4M pre-existing captioned data items. We use 13 different evaluations to demonstrate the value of each data source. Due to limitations with existing benchmarks, we further introduce the first high-quality, consensus-annotated zero-shot classification benchmark between audio and PCs. In contrast to related work, we will open-source our code, model weights, and datasets.

EBind: a practical approach to space binding

TL;DR

EBind tackles the problem of binding embedding spaces across text, image, video, audio, and point clouds with a resource-efficient, data-centric approach. It uses a simple architecture with frozen encoders per modality and small MLP projections, trained through a three-tier data strategy that combines auto-paired quintuplets, human-verified triples, and open-captioned data. With a compact 1.8B-parameter model, EBind achieves state-of-the-art performance on 13 benchmarks and introduces EShot, a high-quality zero-shot PC–audio evaluation benchmark, all while enabling single-GPU training in hours and releasing code and data openly. The work emphasizes data quality and practical training efficiency, offering a scalable path toward accessible, multi-modal binding and broad applicability in retrieval and cross-modal understanding.

Abstract

We simplify space binding by focusing on two core components, a single encoder per modality and high-quality data; enabling training state-of-the-art models on a single GPU in a few hours as opposed to multiple days. We present EBind, an Easy, data-centric, and parameter-efficient method to Bind the embedding spaces of multiple contrastive models. We demonstrate that a simple 1.8B-parameter image-text-video-audio-3D model can outperform models 4 to 17x the size. The key to achieving this is a carefully curated dataset of three complementary data sources: i) 6.7M fully-automated multimodal quintuples sourced via SOTA retrieval models, ii) 1M diverse, semi-automated triples annotated by humans as negative, partial, or positive matches, and iii) 3.4M pre-existing captioned data items. We use 13 different evaluations to demonstrate the value of each data source. Due to limitations with existing benchmarks, we further introduce the first high-quality, consensus-annotated zero-shot classification benchmark between audio and PCs. In contrast to related work, we will open-source our code, model weights, and datasets.

Paper Structure

This paper contains 32 sections, 3 equations, 6 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Current Limitations
  3. Our Contributions: EBind.
  4. Related Work
  5. Model Composition and Training Algorithms
  6. Data for Multimodal Alignment
  7. Open Dataset
  8. Open Dataset
  9. Training Datasets
  10. Split $1$: Automatically Paired 5-tuples (6.7M)
  11. Split $2$: Human verified triples (1M)
  12. Split $3$: Open-source captioned datasets (3.4M)
  13. Zero-Shot PC--Audio Evaluation Dataset
  14. Model and Training
  15. Evaluation
  16. ...and 17 more sections

Figures (6)

  • Figure 1: Overview of the EBind approach and results.
  • Figure 2: A top-1 retrieval example from EBind-P3.
  • Figure 3: Zero-shot PC classification post segmentation.
  • Figure 4: Examples used in annotator instructions in an audio-caption-to-image pairing project
  • Figure 5: Examples used in annotator instructions in an Audio-caption-to-object pairing project
  • ...and 1 more figures