AssistantX: An LLM-Powered Proactive Assistant in Collaborative Human-Populated Environment

TL;DR

This work addresses the limited natural-language and proactive collaboration capabilities of service robots in human-populated spaces. It introduces AssistantX, a proactive, LLM-powered assistant built on the PPDR4X four-agent framework (Perception, Planning, Decision, Reflection) with a shared Memory Unit to coordinate cyber and real-world tasks. Through a 210-task dataset and extensive simulations plus a real-office deployment, the approach demonstrates reactive responsiveness, adaptive strategy adjustment, and proactive human collaboration, outperforming several baselines. The results highlight the potential of cyber-physical integration for real-world productivity, operability, and scalable human-robot collaboration in office environments.

Abstract

Current service robots suffer from limited natural language communication abilities, heavy reliance on predefined commands, ongoing human intervention, and, most notably, a lack of proactive collaboration awareness in human-populated environments. This results in narrow applicability and low utility. In this paper, we introduce AssistantX, an LLM-powered proactive assistant designed for autonomous operation in realworld scenarios with high accuracy. AssistantX employs a multi-agent framework consisting of 4 specialized LLM agents, each dedicated to perception, planning, decision-making, and reflective review, facilitating advanced inference capabilities and comprehensive collaboration awareness, much like a human assistant by your side. We built a dataset of 210 real-world tasks to validate AssistantX, which includes instruction content and status information on whether relevant personnel are available. Extensive experiments were conducted in both text-based simulations and a real office environment over the course of a month and a half. Our experiments demonstrate the effectiveness of the proposed framework, showing that AssistantX can reactively respond to user instructions, actively adjust strategies to adapt to contingencies, and proactively seek assistance from humans to ensure successful task completion. More details and videos can be found at https://assistantx-agent.github.io/AssistantX/.

Figures (10)

• Figure 1: AssistantX overcomes the limitations of existing service robots and virtual assistants, seamlessly integrating physical and virtual actions to meet human needs.
• Figure 2: AssistantX proficiently generates both cyber tasks $\mathcal{TC}$ and real-world tasks $\mathcal{TR}$, executing them concurrently in a manner akin to a human assistant. This approach enhances efficiency and promotes productivity.
• Figure 3: Overview of the proposed framework.
• Figure 4: Overview of the data stored in Memory Unit.
• Figure 5: An illustration of the inputs and outputs of PPDR agents, showing how they collaborate to determine the next move after the previous task, with all agents communicating in natural language, ensuring logical consistency and interpretability.