LLM-based Multi-Agent Blackboard System for Information Discovery in Data Science

TL;DR

The paper tackles the data-discovery bottleneck in large, heterogeneous data lakes when applying LLMs to data science. It introduces a blackboard-based multi-agent system where a central main agent posts information needs to a shared board and autonomous helper agents decide whether to contribute, removing the requirement for a rigid central task allocator. Across KramaBench, DS-Bench, and DA-Code, the approach yields 13%–57% relative improvements in end-to-end task success and up to 9% gains in data-discovery F1 over strong baselines, with both proprietary and open-source LLMs showing improvements. These findings demonstrate a scalable, flexible design for autonomous data-discovery in data science workflows and suggest broad applicability of the blackboard paradigm to complex multi-agent AI systems.

Abstract

The rapid advancement of Large Language Models (LLMs) has opened new opportunities in data science, yet their practical deployment is often constrained by the challenge of discovering relevant data within large heterogeneous data lakes. Existing methods struggle with this: single-agent systems are quickly overwhelmed by large, heterogeneous files in the large data lakes, while multi-agent systems designed based on a master-slave paradigm depend on a rigid central controller for task allocation that requires precise knowledge of each sub-agent's capabilities. To address these limitations, we propose a novel multi-agent communication paradigm inspired by the blackboard architecture for traditional AI models. In this framework, a central agent posts requests to a shared blackboard, and autonomous subordinate agents -- either responsible for a partition of the data lake or general information retrieval -- volunteer to respond based on their capabilities. This design improves scalability and flexibility by eliminating the need for a central coordinator to have prior knowledge of all sub-agents' expertise. We evaluate our method on three benchmarks that require explicit data discovery: KramaBench and modified versions of DS-Bench and DA-Code to incorporate data discovery. Experimental results demonstrate that the blackboard architecture substantially outperforms baselines, including RAG and the master-slave multi-agent paradigm, achieving between 13% to 57% relative improvement in end-to-end task success and up to a 9% relative gain in F1 score for data discovery over the best-performing baselines across both proprietary and open-source LLMs. Our findings establish the blackboard paradigm as a scalable and generalizable communication framework for multi-agent systems.

Figures (16)

• Figure 1: Overview of the blackboard multi-agent system for information discovery in data science. In this framework, the main agent does not assign tasks to subordinate agents. Instead, it posts requests to the blackboard, and subordinate agents autonomously decide whether to respond based on their expertise. The main agent then uses the responses to the request to solve the given task.
• Figure 2: Performance of Blackboard System w/ and w/o search agent (Gemini 2.5 Pro).
• Figure 3: Performance of Blackboard System with various maximum actions by the main agent.
• Figure 4: Evaluation prompt used for DS-Bench dataset using LLM as the judge.
• Figure 5: Prompt used by for clustering the files in data lakes into partitions.