Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Lightweight Multi-Expert Generative Language Model System for Engineering Information and Knowledge Extraction

Bogdan Bogachov, Yaoyao Fiona Zhao

TL;DR

The paper addresses the high cost and hallucination risks of adapting large language models to engineering domains. It introduces the Small Language Graph (SLG), a graph-structured system of LoRA-finetuned, domain-specific expert nodes connected via an orchestrator (LangGraph) to route queries efficiently. Empirical results show SLG achieves approximately a 3x improvement in Exact Match over a larger stand-alone model and requires about 1.7x less fine-tuning time on a single RTX 4090, with potential for distributed, resource-light AI. The approach suggests a viable path for small to medium engineering firms to deploy AI locally and highlights opportunities for scalable, decentralized AI architectures that reduce reliance on expansive compute clusters.

Abstract

Despite recent advancements in domain adaptation techniques for large language models, these methods remain computationally intensive, and the resulting models can still exhibit hallucination issues. Most existing adaptation methods do not prioritize reducing the computational resources required for fine-tuning and inference of language models. Hallucination issues have gradually decreased with each new model release. However, they remain prevalent in engineering contexts, where generating well-structured text with minimal errors and inconsistencies is critical. This work introduces a novel approach called the Small Language Graph (SLG), which is a lightweight adaptation solution designed to address the two key challenges outlined above. The system is structured in the form of a graph, where each node represents a lightweight expert - a small language model fine-tuned on specific and concise texts. The results of this study have shown that SLG was able to surpass conventional fine-tuning methods on the Exact Match metric by 3 times. Additionally, the fine-tuning process was 1.7 times faster compared to that of a larger stand-alone language model. These findings introduce a potential for small to medium-sized engineering companies to confidently use generative AI technologies, such as LLMs, without the necessity to invest in expensive computational resources. Also, the graph architecture and the small size of expert nodes offer a possible opportunity for distributed AI systems, thus potentially diverting the global need for expensive centralized compute clusters.

A Lightweight Multi-Expert Generative Language Model System for Engineering Information and Knowledge Extraction

TL;DR

The paper addresses the high cost and hallucination risks of adapting large language models to engineering domains. It introduces the Small Language Graph (SLG), a graph-structured system of LoRA-finetuned, domain-specific expert nodes connected via an orchestrator (LangGraph) to route queries efficiently. Empirical results show SLG achieves approximately a 3x improvement in Exact Match over a larger stand-alone model and requires about 1.7x less fine-tuning time on a single RTX 4090, with potential for distributed, resource-light AI. The approach suggests a viable path for small to medium engineering firms to deploy AI locally and highlights opportunities for scalable, decentralized AI architectures that reduce reliance on expansive compute clusters.

Abstract

Despite recent advancements in domain adaptation techniques for large language models, these methods remain computationally intensive, and the resulting models can still exhibit hallucination issues. Most existing adaptation methods do not prioritize reducing the computational resources required for fine-tuning and inference of language models. Hallucination issues have gradually decreased with each new model release. However, they remain prevalent in engineering contexts, where generating well-structured text with minimal errors and inconsistencies is critical. This work introduces a novel approach called the Small Language Graph (SLG), which is a lightweight adaptation solution designed to address the two key challenges outlined above. The system is structured in the form of a graph, where each node represents a lightweight expert - a small language model fine-tuned on specific and concise texts. The results of this study have shown that SLG was able to surpass conventional fine-tuning methods on the Exact Match metric by 3 times. Additionally, the fine-tuning process was 1.7 times faster compared to that of a larger stand-alone language model. These findings introduce a potential for small to medium-sized engineering companies to confidently use generative AI technologies, such as LLMs, without the necessity to invest in expensive computational resources. Also, the graph architecture and the small size of expert nodes offer a possible opportunity for distributed AI systems, thus potentially diverting the global need for expensive centralized compute clusters.

Paper Structure

This paper contains 16 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Prompt engineering
  4. Fine-tuning
  5. RAG systems
  6. LLMs in engineering
  7. Methodology
  8. Dataset
  9. SLG
  10. Experiments
  11. Experimentation setup
  12. Fine-tuning strategy
  13. Results
  14. Limitations and future work
  15. Conclusions and discussion
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Data overlapping illustration.
  • Figure 2: Schematic representation of isolated training data.
  • Figure 3: Small Language Graph.
  • Figure 4: Experiment Charts.