Predicting Routine Object Usage for Proactive Robot Assistance

TL;DR

This work tackles long-horizon proactive robot assistance by proposing SLaTe-PRO, a sequential latent model that fuses object-relational scene graphs with activity information to predict future object relocations. It advances prior work by conditioning predictions on past history and by incorporating an interactive query mechanism governed by information gain to address inherently stochastic human behavior. The approach yields substantive improvements over STOT on the augmented HOMER+ dataset, achieving $F1$ scores of up to $0.60$ with user queries, and demonstrates practical robot validation with a fully autonomous Stretch robot. The work also introduces HOMER+ to better capture real-world routine stochasticity, analyzes the impact of behavioral consistency, and discusses limitations and avenues for continual learning and personalization.

Abstract

Proactivity in robot assistance refers to the robot's ability to anticipate user needs and perform assistive actions without explicit requests. This requires understanding user routines, predicting consistent activities, and actively seeking information to predict inconsistent behaviors. We propose SLaTe-PRO (Sequential Latent Temporal model for Predicting Routine Object usage), which improves upon prior state-of-the-art by combining object and user action information, and conditioning object usage predictions on past history. Additionally, we find some human behavior to be inherently stochastic and lacking in contextual cues that the robot can use for proactive assistance. To address such cases, we introduce an interactive query mechanism that can be used to ask queries about the user's intended activities and object use to improve prediction. We evaluate our approach on longitudinal data from three households, spanning 24 activity classes. SLaTe-PRO performance raises the F1 score metric to 0.57 without queries, and 0.60 with user queries, over a score of 0.43 from prior work. We additionally present a case study with a fully autonomous household robot.

Figures (6)

• Figure 1: SLaTe-PRO consists of transformer-based encoder $f_{enc}$ and graph neural network based decoder $f_{dec}$ for object observations $G_t$, learned embeddings $g_{enc}$ and MLP-based decoder $g_{dec}$ for activity labels $a_t$, and a transformer-based predictive model $h$ over latent space $X_t$. These models together learn to predict the object arrangement and activity label at future time-steps. The variables in grey represent observed variables
• Figure 2: (a) SLaTe-PRO outperforms STOT with no activity labels, and steadily improves as more activities become available. (b) With 100% activity labels, SLaTe-PRO outperforms STOT across varying proactivity $\delta$-s
• Figure 3: A steady drop in performance is observed across all methods from more consistent object usage to less.
• Figure 4: (b) Performance gains from queries are most significant for the less consistent object usages. (a) They also improve performance across the entire dataset. '+F' denotes using active queries with the method.
• Figure 5: (a) A Stretch robot assisting proactively with a user's morning routine, by acting on most likely predictions. (b) These tend to fail when the user chooses a less-frequent variation. (c) By querying the user's about their intent, the robot is able to assist with different variations.