Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Reality

TL;DR

This work addresses the challenge of enabling fine-grained, multimodal training in mixed reality by proposing an autonomous workflow that integrates AI agents into MR environments. The system combines a cerebral language agent with memory and planning and a vision-language agent to ground actions in multimodal context, demonstrated within a LEGO brick assembly scenario. A new LEGO-MRTA dataset is synthesized to ground instruction manuals, conversations, XR responses, and vision questions, enabling robust benchmarking of open-source LLMs with and without PEFT via LoRA. The study shows that fine-tuning on LEGO-MRTA improves performance across models and highlights tradeoffs between overlap and informativeness, underscoring the dataset and workflow as a resource for advancing AI assisted MR training and human computer interaction research.

Abstract

Autonomous artificial intelligence (AI) agents have emerged as promising protocols for automatically understanding the language-based environment, particularly with the exponential development of large language models (LLMs). However, a fine-grained, comprehensive understanding of multimodal environments remains under-explored. This work designs an autonomous workflow tailored for integrating AI agents seamlessly into extended reality (XR) applications for fine-grained training. We present a demonstration of a multimodal fine-grained training assistant for LEGO brick assembly in a pilot XR environment. Specifically, we design a cerebral language agent that integrates LLM with memory, planning, and interaction with XR tools and a vision-language agent, enabling agents to decide their actions based on past experiences. Furthermore, we introduce LEGO-MRTA, a multimodal fine-grained assembly dialogue dataset synthesized automatically in the workflow served by a commercial LLM. This dataset comprises multimodal instruction manuals, conversations, XR responses, and vision question answering. Last, we present several prevailing open-resource LLMs as benchmarks, assessing their performance with and without fine-tuning on the proposed dataset. We anticipate that the broader impact of this workflow will advance the development of smarter assistants for seamless user interaction in XR environments, fostering research in both AI and HCI communities.

Figures (8)

• Figure 1: Examples of fine-grained assembly in MR systems.
• Figure 2: The proposed autonomous workflow, involving an AI agent interacting with an MR application. The AI agent comprises a core cerebral language agent, which interacts with a vision-language agent to interpret the multimodal context into metadata, which can be utilized by the cerebral language agent iteratively. The MR application interacts with AI agents by serving functions as external tools.
• Figure 3: An example of LEGO instruction manual. It consists of a summary section at the beginning followed by three sequential instruction steps. Each step includes textual instructions paired with corresponding images to guide the assembly process.
• Figure 4: An example of the generated conversation (left) and the grounding step instructions (right).
• Figure 5: An example of vision-language pair (lower) and the grounding step instructions (upper).