MVSA-Net: Multi-View State-Action Recognition for Robust and Deployable Trajectory Generation

TL;DR

MVSA-Net addresses occlusion in Learn-from-Observation by fusing multiple synchronized RGB-D streams through a gating-network-based mixture-of-experts architecture to perform state and action recognition. It employs a 5-frame temporal context with per-view CNNs and GRUs, plus separate state and action branches whose outputs are fused to produce robust trajectories for imitation or inverse RL. The approach yields higher state-action recognition accuracy than single-view baselines across onion-sorting and patroller-attacker domains, and demonstrates resilience to noise and lighting variations, enabling more reliable deployable trajectory generation. These results indicate that multi-view fusion with dynamic view weighting significantly improves LfO robustness and opens the door to broader real-world robotic deployment.

Abstract

The learn-from-observation (LfO) paradigm is a human-inspired mode for a robot to learn to perform a task simply by watching it being performed. LfO can facilitate robot integration on factory floors by minimizing disruption and reducing tedious programming. A key component of the LfO pipeline is a transformation of the depth camera frames to the corresponding task state and action pairs, which are then relayed to learning techniques such as imitation or inverse reinforcement learning for understanding the task parameters. While several existing computer vision models analyze videos for activity recognition, SA-Net specifically targets robotic LfO from RGB-D data. However, SA-Net and many other models analyze frame data captured from a single viewpoint. Their analysis is therefore highly sensitive to occlusions of the observed task, which are frequent in deployments. An obvious way of reducing occlusions is to simultaneously observe the task from multiple viewpoints and synchronously fuse the multiple streams in the model. Toward this, we present multi-view SA-Net, which generalizes the SA-Net model to allow the perception of multiple viewpoints of the task activity, integrate them, and better recognize the state and action in each frame. Performance evaluations on two distinct domains establish that MVSA-Net recognizes the state-action pairs under occlusion more accurately compared to single-view MVSA-Net and other baselines. Our ablation studies further evaluate its performance under different ambient conditions and establish the contribution of the architecture components. As such, MVSA-Net offers a significantly more robust and deployable state-action trajectory generation compared to previous methods.

Figures (7)

• Figure 1: An overview of how to use MVSA-Net for trajectory generation in a custom domain (onion-sorting) having three input RGB-D streams. First, synchronized RGB-D frames are annotated and used to train the state and action classifiers. Later, unseen synchronized RGB-D frames can be fed to the trained MVSA-Net model to predict the state parameters and the expert's action. In the above example, the expert is about to place a blemished onion in a cardboard box (bin) which is located behind the conveyor belt.
• Figure 2: An overview of the MVSA-Net architecture. The network jointly predicts the state and action of an expert using multiple heterogeneous synchronized RGB-D frames. The so-called RGB-D frames are processed through a deep convolutional and recurrent neural network, which predicts both the state labels and the expert's actions. YOLO-v5 contributes to the network's masking paradigm and aids the decision module in achieving more accurate state predictions. The masking paradigm functions such that when it detects more than one expert in a frame, it automatically converts the input frame into multiple frames, each containing only a single expert.
• Figure 3: A comprehensive example of how the gating network consolidates the results of two distinct classifiers into a solitary output value within the onion sorting domain for recognizing the onion location. The onion location comprises four possibilities, namely $at\_home$, $on\_conveyor$, $in\_front$, and $at\_bin$. Each of these states corresponds to an index ranging from 0 to 3, respectively. As depicted in the diagram, the gating network recognizes that classifier 1 provides more accurate predictions in these scenarios and assigns it a higher weight.
• Figure 4: Multiple viewpoints of onion sorting: (a) front, (b) side, and (c) top-down views.
• Figure 5: Pre-defined spatial regions for the onion and end-effector locations from the front-view camera. In this example, onion and end-effector locations are both in the $in\_front$ region. Furthermore, the onion status is $unblemished$ and the action is $inspect$.