Optimizing Multitask Industrial Processes with Predictive Action Guidance

TL;DR

This work tackles real-time egocentric action anticipation in dynamic industrial environments by coupling a Multi-Modal Transformer Fusion and Recurrent Units (MMTF-RU) with an Operator Action Monitoring Unit (OAMU) to provide proactive guidance and anomaly prevention. The MMTF-RU fuses multimodal signals through a Cross-Modality Fusion Block and uses a GRU decoder to predict the next action, verb, and noun, while the OAMU leverages a Markov-chain reference graph and an entropy-informed score to detect deviations and suggest corrective steps. A novel Time-Weighted Sequence Accuracy (TWSA) metric assesses operator efficiency and adherence to optimal task sequences. Validations on Meccano and EPIC-Kitchens-55 show state-of-the-art or competitive performance for action anticipation, with robust operator guidance and anomaly prevention that enhance reliability and efficiency in industrial assembly workflows.

Abstract

Monitoring complex assembly processes is critical for maintaining productivity and ensuring compliance with assembly standards. However, variability in human actions and subjective task preferences complicate accurate task anticipation and guidance. To address these challenges, we introduce the Multi-Modal Transformer Fusion and Recurrent Units (MMTFRU) Network for egocentric activity anticipation, utilizing multimodal fusion to improve prediction accuracy. Integrated with the Operator Action Monitoring Unit (OAMU), the system provides proactive operator guidance, preventing deviations in the assembly process. OAMU employs two strategies: (1) Top-5 MMTF-RU predictions, combined with a reference graph and an action dictionary, for next-step recommendations; and (2) Top-1 MMTF-RU predictions, integrated with a reference graph, for detecting sequence deviations and predicting anomaly scores via an entropy-informed confidence mechanism. We also introduce Time-Weighted Sequence Accuracy (TWSA) to evaluate operator efficiency and ensure timely task completion. Our approach is validated on the industrial Meccano dataset and the largescale EPIC-Kitchens-55 dataset, demonstrating its effectiveness in dynamic environments.

Figures (10)

• Figure 1: Egocentric activity anticipation: Predicting future actions using the MMTF-RU framework, which determines the next action start time $t_s$ after an anticipation interval $\tau_a$, based on the observation time $\tau_o$.
• Figure 2: Overview of the collaborative assembly workspace. The setup includes (1) an operator's egocentric view and gaze input to the MMTF-RU model, (2) real-time visual feedback for guidance and anomaly alerts, (3) a robotic arm assisting with tasks, and (4) tools and components on the workbench. The MMTF-RU, integrated with OAMU and a knowledge base, provides next-action guidance for efficient assembly operations.
• Figure 3: The architecture of the proposed MMTF-RU framework. Input video features are extracted via a TSN ref35, resulting modality-specific features ($\bm{f}{o}^{0}$, $\bm{f}{h}^{0}$, $\bm{f}{g}^{0}$). These, along with positional embeddings (PE), are processed by transformer encoders to produce transformed features ($\bm{f}{o}^{l}$, $\bm{f}{h}^{l}$, $\bm{f}{g}^{l}$). The CMFB integrates features across modalities, and GRUs generate temporal decoder features based on anticipation time $\tau_a$. Finally, these features are classified to predict the next action, verb, or noun ($\hat{Y}$).
• Figure 4: (a) Reference graph and (b) transition heatmaps in the Meccano dataset.
• Figure 5: Visualization of Top-1 action anticipation results for (a) Meccano and (b) EPIC-Kitchens-55 datasets. Ground truth (GT) actions are highlighted in blue, correct predictions (PT) in green, and incorrect predictions in red.