Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Graph-Guided Multi-Agent Distillation for Reliable Industrial Question Answering with Datasets

Jiqun Pan, Zhenke Duan, Jiani Tu, Anzhi Cheng, Yanqing Wang

TL;DR

This work tackles the need for safe, reliable industrial QA by introducing KG-MASD, a knowledge-graph-guided multi-agent distillation framework. It models distillation as a Markov decision process with self-generated KG priors, grounding collaborative reasoning and enabling verifiable outputs through a verifier loop and GraphRAG-derived knowledge graphs. The approach yields theoretical benefits in convergence and variance reduction, demonstrates strong empirical gains over both MAS-based and single-model baselines on a vertical industrial QA dataset, and highlights the importance of data credibility and completeness for KG-guided transfer. By enabling efficient edge deployment with LoRA and providing an open industrial QA dataset and code, KG-MASD advances trustworthy AI for high-stakes industrial scenarios.

Abstract

Industrial question-answering (QA) systems require higher safety and reliability than general-purpose dialogue models, as errors in high-risk scenarios such as equipment fault diagnosis can have severe consequences. Although multi-agent large language models enhance reasoning depth, they suffer from uncontrolled iterations and unverifiable outputs, and conventional distillation methods struggle to transfer collaborative reasoning capabilities to lightweight, deployable student models. To address these challenges, we propose Knowledge Graph-guided Multi-Agent System Distillation (KG-MASD). Our approach formulates distillation as a Markov Decision Process and incorporates a knowledge graph as a verifiable structured prior to enrich state representation and ensure convergence. By integrating collaborative reasoning with knowledge grounding, KG-MASD generates high-confidence instruction-tuning data and jointly distills reasoning depth and verifiability into compact student models suitable for edge deployment. Experiments on an industrial QA dataset show that KG-MASD improves accuracy by 2.4 per cent to 20.1 per cent over baselines and significantly enhances reliability, enabling trustworthy AI deployment in safety-critical industrial scenarios. Code and data are available at https://github.com/erwinmsmith/KG-MAD/.

Knowledge Graph-Guided Multi-Agent Distillation for Reliable Industrial Question Answering with Datasets

TL;DR

This work tackles the need for safe, reliable industrial QA by introducing KG-MASD, a knowledge-graph-guided multi-agent distillation framework. It models distillation as a Markov decision process with self-generated KG priors, grounding collaborative reasoning and enabling verifiable outputs through a verifier loop and GraphRAG-derived knowledge graphs. The approach yields theoretical benefits in convergence and variance reduction, demonstrates strong empirical gains over both MAS-based and single-model baselines on a vertical industrial QA dataset, and highlights the importance of data credibility and completeness for KG-guided transfer. By enabling efficient edge deployment with LoRA and providing an open industrial QA dataset and code, KG-MASD advances trustworthy AI for high-stakes industrial scenarios.

Abstract

Industrial question-answering (QA) systems require higher safety and reliability than general-purpose dialogue models, as errors in high-risk scenarios such as equipment fault diagnosis can have severe consequences. Although multi-agent large language models enhance reasoning depth, they suffer from uncontrolled iterations and unverifiable outputs, and conventional distillation methods struggle to transfer collaborative reasoning capabilities to lightweight, deployable student models. To address these challenges, we propose Knowledge Graph-guided Multi-Agent System Distillation (KG-MASD). Our approach formulates distillation as a Markov Decision Process and incorporates a knowledge graph as a verifiable structured prior to enrich state representation and ensure convergence. By integrating collaborative reasoning with knowledge grounding, KG-MASD generates high-confidence instruction-tuning data and jointly distills reasoning depth and verifiability into compact student models suitable for edge deployment. Experiments on an industrial QA dataset show that KG-MASD improves accuracy by 2.4 per cent to 20.1 per cent over baselines and significantly enhances reliability, enabling trustworthy AI deployment in safety-critical industrial scenarios. Code and data are available at https://github.com/erwinmsmith/KG-MAD/.

Paper Structure

This paper contains 44 sections, 2 theorems, 19 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Distillation
  4. Multi-agent Systems
  5. Industrial Question-Answering Knowledge
  6. Dataset and Augmentation
  7. Dataset Compilation and Processing Workflow
  8. Dataset Challenges
  9. Benchmark Test
  10. Dataset Augmentation Directions
  11. Methodology
  12. Overview
  13. Multi-agent System Design
  14. Theoretical Foundation: MDP Modeling with Self-guided Priors
  15. Notation and basic objects
  16. ...and 29 more sections

Key Result

Theorem 4.1

Assume $\gamma_2>\gamma_1$ and $S^{\gamma_2}$ Blackwell-dominates $S^{\gamma_1}$. Let the one-step reward be a proper scoring rule for predicting $Y$ from $S^\gamma$ (e.g., eq:reward-examples). Then Proof. For a proper scoring rule, the Bayes act using observation $S^\gamma$ is the posterior $p^\ast(y\mid S^\gamma)$, and the Bayes risk equals a proper functional of the posterior. For the log-scor

Figures (12)

  • Figure 1: (a) Challenges faced by edge-side models in knowledge graph extraction and (b) QA scenarios, highlighting the necessity of credibility-aware distillation in industrial domains.
  • Figure 2: The proportions of the eight manually extracted thematic categories are as follows: Transportation accounts for 6.5%, Health for 2.63%, General for 39.68%, Environment for 2.41%, Equipment for 18.42%, Production for 5.31%, Electricity for 20.17%, and Disaster Prevention for 4%.
  • Figure 3: End-to-end data processing workflow. Raw industrial data from three sources—Entity Mappings, Human QA pairs, and GPT-generated QA—are preprocessed, normalized, and transformed into structured triples. Outputs are stored in Excel (for human auditing) and JSON (for automated evaluation), with quality assessed via BLEU, ROUGE, and human-in-the-loop validation.
  • Figure 4: It illustrates the overall process of extracting the Global Knowledge Graph (GKG) from raw data using GraphRAG technology, and conducting entity and relation extraction, local knowledge graph generation, and instruction fine-tuning via a Multi-agent system.
  • Figure 5: As time progresses and the number of iterations increases, the KG-MASD system achieves rapid self-stabilization. The Verifier also stabilizes accordingly.
  • ...and 7 more figures

Theorems & Definitions (2)

  • Theorem 4.1: Reward Monotonicity via Information Dominance
  • Theorem 4.3: Faster transfer via variance reduction