Real-Time Manipulation Action Recognition with a Factorized Graph Sequence Encoder

TL;DR

This work addresses real-time manipulation action recognition by introducing the Factorized Graph Sequence Encoder (FGSE), which decouples spatial graph encoding from temporal sequence modeling. FGSE uses a Transformer-based Graph Encoder and a separate Sequence Encoder with a parameter-free Hand Pooling operation to extract hand-focused embeddings, allowing temporal scalability without deepening the graph network. A sliding-window with majority voting combines predictions, achieving state-of-the-art real-time performance on Bimacs and CoAx while remaining highly parameter-efficient (~269k parameters) and fast (~66 FPS). The approach outperforms RGB-based baselines on object-centric datasets, validating the advantage of scene-graph representations for generalizable manipulation understanding in HRC contexts. Future work aims to extend to skeleton-based tasks and to better handle noisy graph estimates by incorporating estimation confidence.

Abstract

Recognition of human manipulation actions in real-time is essential for safe and effective human-robot interaction and collaboration. The challenge lies in developing a model that is both lightweight enough for real-time execution and capable of generalization. While some existing methods in the literature can run in real-time, they struggle with temporal scalability, i.e., they fail to adapt to long-duration manipulations effectively. To address this, leveraging the generalizable scene graph representations, we propose a new Factorized Graph Sequence Encoder network that not only runs in real-time but also scales effectively in the temporal dimension, thanks to its factorized encoder architecture. Additionally, we introduce Hand Pooling operation, a simple pooling operation for more focused extraction of the graph-level embeddings. Our model outperforms the previous state-of-the-art real-time approach, achieving a 14.3\% and 5.6\% improvement in F1-macro score on the KIT Bimanual Action (Bimacs) Dataset and Collaborative Action (CoAx) Dataset, respectively. Moreover, we conduct an extensive ablation study to validate our network design choices. Finally, we compare our model with its architecturally similar RGB-based model on the Bimacs dataset and show the limitations of this model in contrast to ours on such an object-centric manipulation dataset.

Figures (3)

• Figure 1: The proposed Factorized Graph Sequence Encoder (FGSE) network.
• Figure 2: Majority voting to combine the shifted predictions. Starting from the top row, as the window slides along the temporal axis (x-axis), a new set of predictions is generated. The window size (W) is 5 in here. Each colored box represents a different predicted manipulation label.
• Figure 3: An example run from the test set of Bimacs dreher, in which a person pours water from a bottle into a cup, and then drinks it. The top three rows show the ground-truth labels, predictions, and vote count in majority voting for the left hand, and the next three rows correspond to the right hand. Each color represents different actions: idle, approach, lift, hold, pour, place, retreat, drink.