COOPERA: Continual Open-Ended Human-Robot Assistance

TL;DR

COOPERA tackles open-ended, continual human-robot collaboration by modeling simulated humans with enduring traits and evolving intentions, enabling robots to personalize assistance over multiple days. The framework merges LLM-driven human simulation, VLM-based assistive reasoning, and a feedback loop to learn correlations between human traits, temporal context, and goals. A Habitat-based benchmark and a full pipeline for day-by-day adaptation demonstrate notable within-day and across-day improvements, with ablations confirming the importance of trait inference and temporal dependencies. The work advances long-horizon HRC research and opens avenues for real-world, proactive robot assistance in dynamic household environments.

Abstract

To understand and collaborate with humans, robots must account for individual human traits, habits, and activities over time. However, most robotic assistants lack these abilities, as they primarily focus on predefined tasks in structured environments and lack a human model to learn from. This work introduces COOPERA, a novel framework for COntinual, OPen-Ended human-Robot Assistance, where simulated humans, driven by psychological traits and long-term intentions, interact with robots in complex environments. By integrating continuous human feedback, our framework, for the first time, enables the study of long-term, open-ended human-robot collaboration (HRC) in different collaborative tasks across various time-scales. Within COOPERA, we introduce a benchmark and an approach to personalize the robot's collaborative actions by learning human traits and context-dependent intents. Experiments validate the extent to which our simulated humans reflect realistic human behaviors and demonstrate the value of inferring and personalizing to human intents for open-ended and long-term HRC. Project Page: https://dannymcy.github.io/coopera/

Figures (13)

• Figure 1: Continual human-robot collaboration for open-ended tasks over multiple days. Our framework COOPERA entails an approach to simulate traits-driven humans with long-term, whole-day behaviors within robot simulation platform, enabling the first study of long-term, open-ended human-robot collaboration. We also introduce a benchmark and a method for the robot to personalize collaboration in such continual, open-ended settings by learning human traits and context-dependent intents over time.
• Figure 2: COOPERA: Continual, open-ended human-robot collaboration framework. The LLM-powered human proposes whole-day intentions and tasks, executed in the environment. As the robot observes the human actions, it predicts a set of tasks to assist them. After each day, the human provides feedback to the robot, enabling the robot to improve for subsequent days.
• Figure 3: Human Simulation Pipeline. We seed the human-LLM with an extended profile. At each time of day, the human proposes an intention and decomposes it into tasks, aligning with profile traits and temporal dependence on intention/task history. LLM inputs are optimized with Memory Retrieval and Search, and robustness is enhanced via two rounds of Reflexion. This pipeline generates continuous, whole-day intentions and tasks executed in the environment with expressive whole-body motion. See Appendices \ref{['appendix:Additional Details of Human Simulation']} and \ref{['sec:Prompt Details of the Simulated Human']} for details.
• Figure 4: Our approach for human assistance. We decouple robot task inference into intention and task inference. By chaining VLM and classifier, the robot selects tasks aligned with the human’s traits and temporal context. It maintains a human profile inferred from collaboration history, which, combined with feedback, optimizes the robot-VLM via prompting and the classifiers via supervised learning. See Appendices \ref{['appendix:Additional Details of Building and Evaluating an Assistive Agent']} and \ref{['sec:Prompt Details of the Assistive Agent']} for details.
• Figure 5: Qualitative examples of full-day intentions and tasks proposed by a human with specific human traits and psychometric data.