S-DAG: A Subject-Based Directed Acyclic Graph for Multi-Agent Heterogeneous Reasoning

TL;DR

S-DAG introduces a subject-based directed acyclic graph to tackle heterogeneous reasoning across multiple domains. It combines a Graph Neural Network to generate a pruned subject DAG with a subject-capability profile of multiple domain-specific LLMs to orchestrate graph-structured multi-agent collaboration, enabling efficient, cross-domain reasoning. The approach yields substantial accuracy gains (average ~59.7%) over task-level baselines and remains competitive with large monolithic LLMs, while reducing computation via sparse, directional information flow. The authors curate multi-subject subsets of MMLU-Pro, GPQA, and MedMCQA and provide extensive ablations that highlight the value of the DAG structure, GNN coordination, and subject-aware model selection for practical multi-subject reasoning tasks.

Abstract

Large Language Models (LLMs) have achieved impressive performance in complex reasoning problems. Their effectiveness highly depends on the specific nature of the task, especially the required domain knowledge. Existing approaches, such as mixture-of-experts, typically operate at the task level; they are too coarse to effectively solve the heterogeneous problems involving multiple subjects. This work proposes a novel framework that performs fine-grained analysis at subject level equipped with a designated multi-agent collaboration strategy for addressing heterogeneous problem reasoning. Specifically, given an input query, we first employ a Graph Neural Network to identify the relevant subjects and infer their interdependencies to generate an \textit{Subject-based Directed Acyclic Graph} (S-DAG), where nodes represent subjects and edges encode information flow. Then we profile the LLM models by assigning each model a subject-specific expertise score, and select the top-performing one for matching corresponding subject of the S-DAG. Such subject-model matching enables graph-structured multi-agent collaboration where information flows from the starting model to the ending model over S-DAG. We curate and release multi-subject subsets of standard benchmarks (MMLU-Pro, GPQA, MedMCQA) to better reflect complex, real-world reasoning tasks. Extensive experiments show that our approach significantly outperforms existing task-level model selection and multi-agent collaboration baselines in accuracy and efficiency. These results highlight the effectiveness of subject-aware reasoning and structured collaboration in addressing complex and multi-subject problems.

Figures (5)

• Figure 1: Comparison of the prior method with single agent and the proposed S-DAG approach with multi-agent collaboration. The prior method routes the problem to a single agent based on a coarse domain label, while our S-DAG approach conducts fine-grained subject analysis, identifying multiple relevant domains with associated relevant weights.
• Figure 2: The Overview of the S-DAG Framework. The framework operates in two stages. In Stage 1, the input question is encoded using a BERT encoder, and a Graph Decoder generates the S-DAG, capturing subject dependencies and pruning irrelevant subjects. In Stage 2, expert LLMs are selected based on their subject-specific expertise and organized according to the S-DAG, with directed edges defining the information flow for multi-subject collaborative reasoning.
• Figure 3: Comparison between the Fully Connected Graph and our S-DAG.
• Figure 4: The DAG construction given subjects and their corresponding weights.
• Figure 5: The S-DAG structured multi-agent information flow.