SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World

TL;DR

This work introduces Chain-of-User-Thought (COUT), a novel reasoning paradigm that integrates user personalization into embodied agents. It presents SmartAgent, a two-stage training framework that fuses GUI grounding with explicit and implicit personalization using a Perceiver-Reasoner LVLM backbone, and demonstrates the approach on the SmartSpot benchmark comprising single- and multi-channel GUI tasks. The results show strong performance in embodied perception, underlying user-intent reasoning, and item recommendations, including notable zero-shot capabilities in unseen scenarios. The work highlights the potential of combining structured thought sequences with multimodal GUI interactions to create more personalized and adaptable embodied agents for cyber-world applications.

Abstract

Recent advances in embodied agents with multimodal perception and reasoning capabilities based on large vision-language models (LVLMs), excel in autonomously interacting either real or cyber worlds, helping people make intelligent decisions in complex environments. However, the current works are normally optimized by golden action trajectories or ideal task-oriented solutions toward a definitive goal. This paradigm considers limited user-oriented factors, which could be the reason for their performance reduction in a wide range of personal assistant applications. To address this, we propose Chain-of-User-Thought (COUT), a novel embodied reasoning paradigm that takes a chain of thought from basic action thinking to explicit and implicit personalized preference thought to incorporate personalized factors into autonomous agent learning. To target COUT, we introduce SmartAgent, an agent framework perceiving cyber environments and reasoning personalized requirements as 1) interacting with GUI to access an item pool, 2) generating users' explicit requirements implied by previous actions, and 3) recommending items to fulfill users' implicit requirements. To demonstrate SmartAgent's capabilities, we also create a brand-new dataset SmartSpot that offers a full-stage personalized action-involved environment. To our best knowledge, our work is the first to formulate the COUT process, serving as a preliminary attempt towards embodied personalized agent learning. Our extensive experiments on SmartSpot illuminate SmartAgent's functionality among a series of embodied and personalized sub-tasks. We will release code and data upon paper notification at https://github.com/tsinghua-fib-lab/SmartAgent.

Figures (4)

• Figure 1: The Chain-of-User-Thought (COUT) reasoning paradigm. The red line shows a sequence of GUI actions, while the blue line illustrates our three-step thought process. In Thought #1, according to a user's instruction, an agent performs GUI actions to search for an item pool. In Thought #2 with seeing the pool, the agent reasons underlying requirements behind the original instruction, as implied by the previous actions. In Thought #3, based on the underlying thought, the agent recommends items within the pool to complete the user's instruction. By leveraging user-oriented thoughts, this COUT could enable full-stage embodied personalized capabilities across various information systems.
• Figure 2: The full-stage embodied personalized capabilities of our proposed SmartAgent, range from basic environment cognition to advanced user personal intention reasoning.
• Figure 4: Comparison of methods on SmartSpot. SmartAgent performs comparable and even better with GUI specialist model and general LLM in all scenarios. Notably, in the more complex scenarios TRAVEL1 and TRAVEL2, which involve longer episodes, SmartAgent consistently shows excellent performance across all embodied and personalized metrics.
• Figure 5: Case study for embodied personalized reasoning.