STREAK: Streaming Network for Continual Learning of Object Relocations under Household Context Drifts

TL;DR

STREAK addresses continual learning for domestic robots facing context drifts and non-i.i.d. data by deploying a Spatio-Temporal Streaming Graph Neural Network with regularization and rehearsal. It extends streaming GNNs to model object relocations across multiple households, using a consolidation loss with Fisher Information and a dynamically managed memory buffer to mitigate forgetting while learning new tasks. Evaluations on the HOMER dataset show STREAK retains past knowledge while remaining competitive with an upper-bound model, and it demonstrates practical time and memory efficiency suitable for real-world robot deployment. A real robot demonstration with ARI further validates the approach, illustrating proactive assistance in dynamic home environments and highlighting its potential for scalable long-term human-robot interaction.

Abstract

In real-world settings, robots are expected to assist humans across diverse tasks and still continuously adapt to dynamic changes over time. For example, in domestic environments, robots can proactively help users by fetching needed objects based on learned routines, which they infer by observing how objects move over time. However, data from these interactions are inherently non-independent and non-identically distributed (non-i.i.d.), e.g., a robot assisting multiple users may encounter varying data distributions as individuals follow distinct habits. This creates a challenge: integrating new knowledge without catastrophic forgetting. To address this, we propose STREAK (Spatio Temporal RElocation with Adaptive Knowledge retention), a continual learning framework for real-world robotic learning. It leverages a streaming graph neural network with regularization and rehearsal techniques to mitigate context drifts while retaining past knowledge. Our method is time- and memory-efficient, enabling long-term learning without retraining on all past data, which becomes infeasible as data grows in real-world interactions. We evaluate STREAK on the task of incrementally predicting human routines over 50+ days across different households. Results show that it effectively prevents catastrophic forgetting while maintaining generalization, making it a scalable solution for long-term human-robot interactions.

Figures (4)

• Figure 1: The robot inspects the spatio-temporal dynamics of the objects in two different households.
• Figure 2: Overview of proposed STREAK framework. The robot acquires the graph state through user action observation, learning each household in sequence. For each household, the robot assembles the respective graph state, $G^{k}_{t}$, at a given time step $t$. Finally, it predicts dynamic spatial object relocations according to user routines for each household.
• Figure 3: Evaluation of "Moved Correct" of finetuned_GRAPH and STREAK on the 5 datasets, after the models have been trained sequentially on $D_0 \rightarrow D_4$
• Figure 4: The dimension (number of samples) of $\mathcal{M}_{k}$ across the five learning sessions (blue line), and the estimated size of $\mathcal{M}_{k}$ after 10 learning sessions (orange line) where we considered all the datasets of the same size, equal to the mean of the five existing ones.