Structure-aware Domain Knowledge Injection for Large Language Models

TL;DR

This paper tackles the data-efficiency challenge in adapting foundation LLMs to specialized domains by introducing StructTuning, a structure-aware two-stage framework. It automatically extracts a domain knowledge structure from textbooks and reorganizes training data, then applies Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT) to inject and elicit this knowledge. The approach yields notable improvements on LongBench and MMedBench across multiple models, achieving substantial performance gains with only a fraction of traditional training data (e.g., 0.3% for large gains and ~5% for near-full gains). The results highlight strong scalability and cross-language generalization, with RAG-based baselines offering limited benefits in this setting. Overall, StructTuning demonstrates a promising path to more data-efficient, domain-specific LLMs that better emulate structured human knowledge acquisition.

Abstract

This paper introduces a pioneering methodology, termed StructTuning, to efficiently transform foundation Large Language Models (LLMs) into domain specialists. It significantly reduces the training corpus needs to a mere 5% while achieving an impressive 100% of traditional knowledge injection performance. Motivated by structured human education, we propose a novel two-stage strategy for knowledge injection and alignment: Structure-aware Continual Pre-Training (SCPT) and Structure-aware Supervised Fine-Tuning (SSFT). In the SCPT phase, we automatically extract the domain knowledge taxonomy and reorganize the training corpora, enabling LLMs to effectively link textual segments to targeted knowledge points within the taxonomy. In the SSFT phase, we explicitly prompt models to elucidate the underlying knowledge structure in their outputs, leveraging the structured domain insight to address practical problems. Our ultimate method was extensively evaluated across model architectures and scales on LongBench and MMedBench datasets, demonstrating superior performance against other knowledge injection methods. We also explored our method's scalability across different training corpus sizes, laying the foundation to enhance domain-specific LLMs with better data utilization.

Figures (11)

• Figure 1: Discrepancy between human education and vanilla LLM adaptation. Human students learn structured knowledge through textbooks section by section, with particular exercises on related knowledge points. Traditional LLM adaptation continually pre-trains on data chunks from randomly concatenated text segments, with aimless supervised fine-tuning for conversation alignment. The inherent knowledge structure is ignored.
• Figure 2: Framework for structure-aware knowledge injection. We extract the inherent knowledge structure in the training corpus, and associate training chunks to corresponding knowledge points. Models are continually pre-trained on data chunks in the condition of the knowledge structure, and fine-tuned with supervised QA samples to elicit their learned knowledge to solve knowledge-intensive (KI) and 2- or multi-hop questions in the real world.
• Figure 3: Left: extracted knowledge structure. Right: template to bridge mindmap structure and textual contents.
• Figure 4: QA samples synthesized for SSFT. We instruct Llama3-70B to generate (a) knowledge-intensive and (b) multi-hop questions and derive the diagnosis answers with explicit reasoning.
• Figure 5: An example for structure-aware responses.