How to Build AI Agents by Augmenting LLMs with Codified Human Expert Domain Knowledge? A Software Engineering Framework

TL;DR

This work addresses the expert bottleneck in engineering visualization by proposing a software engineering framework that codifies tacit and explicit domain knowledge into an AI agent capable of autonomous, expert-level visualization generation. It combines a request classifier, Retrieval-Augmented Generation for code, codified expert rules, and visualization design principles within a unified agent architecture, validated across five scenarios in three engineering domains with 12 evaluators, achieving a mean output quality improvement of 206% and expert-level ratings. The results demonstrate that non-experts can produce high-quality, informative visualizations by leveraging domain-guided rules and prompts, reducing reliance on scarce experts and enabling scalable design exploration. The framework’s physics-agnostic design pattern supports cross-domain reuse, suggesting practical impact for industrial visualization tasks and broader automation of expert knowledge capture and application.

Abstract

Critical domain knowledge typically resides with few experts, creating organizational bottlenecks in scalability and decision-making. Non-experts struggle to create effective visualizations, leading to suboptimal insights and diverting expert time. This paper investigates how to capture and embed human domain knowledge into AI agent systems through an industrial case study. We propose a software engineering framework to capture human domain knowledge for engineering AI agents in simulation data visualization by augmenting a Large Language Model (LLM) with a request classifier, Retrieval-Augmented Generation (RAG) system for code generation, codified expert rules, and visualization design principles unified in an agent demonstrating autonomous, reactive, proactive, and social behavior. Evaluation across five scenarios spanning multiple engineering domains with 12 evaluators demonstrates 206% improvement in output quality, with our agent achieving expert-level ratings in all cases versus baseline's poor performance, while maintaining superior code quality with lower variance. Our contributions are: an automated agent-based system for visualization generation and a validated framework for systematically capturing human domain knowledge and codifying tacit expert knowledge into AI agents, demonstrating that non-experts can achieve expert-level outcomes in specialized domains.

Figures (14)

• Figure 1: Software Engineering Framework for Knowledge Codification
• Figure 2: AI agent architecture. User query enters the agent's perception layer (classifier), which autonomously routes to appropriate processing components.
• Figure 3: History plot generated by LLM+RAG
• Figure 4: Human generated history plot
• Figure 5: History plot generated by AI agent