CMDBench: A Benchmark for Coarse-to-fine Multimodal Data Discovery in Compound AI Systems

TL;DR

CMDBench tackles the data discovery challenge faced by compound AI systems operating over siloed enterprise data. It builds a multimodal benchmark by integrating NBA-domain documents, tables, and graphs (via KILT, WikiSQL, and Wikidata) under a unified knowledge source, enabling coarse- and fine-grained evaluation of source routing and data retrieval. Experimental results with LlamaIndex-based CASs reveal substantial gaps: even strong LLMs and retrieval baselines yield notable drops in task accuracy when discovery is used instead of oracle access ($\sim46\%$), and performance gains are uneven across modalities and task difficulty. The work highlights the need for optimized agent-selector and retriever designs and lays out a scalable path for expanding benchmarks to more domains and data types, with practical implications for enterprise data platforms.

Abstract

Compound AI systems (CASs) that employ LLMs as agents to accomplish knowledge-intensive tasks via interactions with tools and data retrievers have garnered significant interest within database and AI communities. While these systems have the potential to supplement typical analysis workflows of data analysts in enterprise data platforms, unfortunately, CASs are subject to the same data discovery challenges that analysts have encountered over the years -- silos of multimodal data sources, created across teams and departments within an organization, make it difficult to identify appropriate data sources for accomplishing the task at hand. Existing data discovery benchmarks do not model such multimodality and multiplicity of data sources. Moreover, benchmarks of CASs prioritize only evaluating end-to-end task performance. To catalyze research on evaluating the data discovery performance of multimodal data retrievers in CASs within a real-world setting, we propose CMDBench, a benchmark modeling the complexity of enterprise data platforms. We adapt existing datasets and benchmarks in open-domain -- from question answering and complex reasoning tasks to natural language querying over structured data -- to evaluate coarse- and fine-grained data discovery and task execution performance. Our experiments reveal the impact of data retriever design on downstream task performance -- a 46% drop in task accuracy on average -- across various modalities, data sources, and task difficulty. The results indicate the need to develop optimization strategies to identify appropriate LLM agents and retrievers for efficient execution of CASs over enterprise data.

Figures (8)

• Figure 1: Each team-specific data source $DS_i$ represents a collection of sources ($S_m \in DS_i$) corresponding to different modalities ($m$) where $m \in \{D, G, T\}$ ($D$ = Document, $G$ = Graph, $T$ = Table). Collections of data $C_{mj} \in S_m$ may be stored in respective DBMSs --- tables in relational DBMS such as Postgres stonebraker1986design, text data in document DBMSs such as MongoDB bradshaw2019mongodb, graphs in property graph DBMSs such as Neo4j miller2013graph.
• Figure 2: A simplified representation of the CMDBench scope and setting with only one data source corresponding to basketball ($DS_b$). The tabular source ($S_T$) have statistics about players and teams, the graph source ($S_G$) contains symbolic knowledge and relationship among concepts, and the document source ($S_D$) has additional contextual information.
• Figure 3: Snapshot of the NBA data source explored in CMDBench showing 1-hop neighborhood of the player Stephen Curry in Wikidata vrandevcic2014wikidata, his Wikipedia biography retrieved from the KILT document collection petroni2021kilt, and a table from WikiSQL zhongSeq2SQL2017 with a list of NBA player appearances in different seasons.
• Figure 4: WikiSQL table corresponding to $Q3$ in Table \ref{['tab:tabular-task']}.
• Figure 5: Schema of the NBA graph extracted from Wikidata.