MQRLD: A Multimodal Data Retrieval Platform with Query-aware Feature Representation and Learned Index Based on Data Lake

TL;DR

The paper tackles the challenge of efficient multimodal data retrieval by integrating transparent data lake storage, a flexible multimodal open API for rich hybrid queries, and a unified, query-aware feature representation pipeline augmented by a high-dimensional learned index. The framework, MQRLD, comprises a data-lake backed backbone, a MOAPI for flexible querying, a two-stage feature embedding/measurement pipeline with a hyperspace transformation and LPGF-based feature enhancement, and a divisive hierarchical index with last-mile training guided by query behavior. Experimental results across diverse real and synthetic datasets show substantial gains in range and KNN queries, robustness to data scale and dimensionality, and strong performance on rich hybrid queries, outperforming both traditional and learned indexes. The work demonstrates that aligning data representation with query workloads and embedding a learned, workload-aware index within a data lake can significantly improve multimodal retrieval in large-scale settings, with implications for data analytics, search, and AI-enabled mining tasks.

Abstract

Multimodal data has become a crucial element in the realm of big data analytics, driving advancements in data exploration, data mining, and empowering artificial intelligence applications. To support high-quality retrieval for these cutting-edge applications, a robust multimodal data retrieval platform should meet the challenges of transparent data storage, rich hybrid queries, effective feature representation, and high query efficiency. However, among the existing platforms, traditional schema-on-write systems, multi-model databases, vector databases, and data lakes, which are the primary options for multimodal data retrieval, make it difficult to fulfill these challenges simultaneously. Therefore, there is an urgent need to develop a more versatile multimodal data retrieval platform to address these issues. In this paper, we introduce a Multimodal Data Retrieval Platform with Query-aware Feature Representation and Learned Index based on Data Lake (MQRLD). It leverages the transparent storage capabilities of data lakes, integrates the multimodal open API to provide a unified interface that supports rich hybrid queries, introduces a query-aware multimodal data feature representation strategy to obtain effective features, and offers high-dimensional learned indexes to optimize data query. We conduct a comparative analysis of the query performance of MQRLD against other methods for rich hybrid queries. Our results underscore the superior efficiency of MQRLD in handling multimodal data retrieval tasks, demonstrating its potential to significantly improve retrieval performance in complex environments. We also clarify some potential concerns in the discussion.

Figures (31)

• Figure 1: Querying in multimodal data, including structured query attributes ("10$-20$" and "0-24 hours") and vector query attributes ("CupColor.jpg" and "CupDescription.mp3").
• Figure 2: MQRLD framework.
• Figure 3: MQRLD workflow.
• Figure 4: Transparent data storage and rich hybrid queries in MQRLD.
• Figure 5: Overview of multimodal data representation, including feature embedding and measurement, feature enhancement, and optimization based on the query-aware mechanism.