Bridging Industrial Expertise and XR with LLM-Powered Conversational Agents

TL;DR

The paper addresses the challenge of knowledge transfer in industrial environments by integrating Retrieval-Augmented Generation (RAG) enhanced LLMs with Extended Reality (XR) to deliver contextual, hands-free expert guidance. It presents a bi-directional architecture where an LLM Chat Engine handles natural language processing and knowledge retrieval, while an XR application provides immersive visualization and voice interaction; a vector-store backed pipeline and dynamic tool orchestration underpin domain-specific retrieval. Key contributions include a Data Injection Pipeline with semantic chunking, a modular Query Router with PdM/XAI/IoT agents, and a performance evaluation showing semantic chunking and Pinecone as top performers for recall and faithfulness. The work demonstrates early industrial use cases such as robotic assembly, smart infrastructure maintenance, and aerospace servicing, highlighting potential improvements in training efficiency, remote assistance, and operational guidance aligned with Industry 5.0 principles.

Abstract

This paper introduces a novel integration of Retrieval-Augmented Generation (RAG) enhanced Large Language Models (LLMs) with Extended Reality (XR) technologies to address knowledge transfer challenges in industrial environments. The proposed system embeds domain-specific industrial knowledge into XR environments through a natural language interface, enabling hands-free, context-aware expert guidance for workers. We present the architecture of the proposed system consisting of an LLM Chat Engine with dynamic tool orchestration and an XR application featuring voice-driven interaction. Performance evaluation of various chunking strategies, embedding models, and vector databases reveals that semantic chunking, balanced embedding models, and efficient vector stores deliver optimal performance for industrial knowledge retrieval. The system's potential is demonstrated through early implementation in multiple industrial use cases, including robotic assembly, smart infrastructure maintenance, and aerospace component servicing. Results indicate potential for enhancing training efficiency, remote assistance capabilities, and operational guidance in alignment with Industry 5.0's human-centric and resilient approach to industrial development.

Figures (7)

• Figure 1: Architectural Overview of RAG-Enhanced LLM with XR Integration for Industrial Environments. The diagram illustrates the bi-directional communication flow between the XR Application and LLM Chat Engine through middleware services, highlighting the document processing pipeline that populates the vector database for knowledge retrieval.
• Figure 2: High-level Architectural View of LLM Chat Engine
• Figure 3: Data and Knowledge Injection Pipeline
• Figure 4: XR Architecture
• Figure 5: Example response from the LLM Chat Engine to a user query.