NSF-MAP: Neurosymbolic Multimodal Fusion for Robust and Interpretable Anomaly Prediction in Assembly Pipelines

TL;DR

This work tackles the challenge of reliable and interpretable anomaly prediction in complex assembly pipelines by proposing NSF-MAP, a neurosymbolic multimodal fusion framework that combines time-series signals with ROI-cropped images. The method extracts time-series latent features via an autoencoder and image features via a pretrained EfficientNet-B0, fusing them at the decision level and enhancing predictions with a dynamic Process Ontology for knowledge infusion. Key contributions include a novel neurosymbolic fusion architecture, two derived multimodal datasets, and ontology-enabled user-level explanations, with ablation studies showing robust gains from transfer learning and knowledge infusion. The approach demonstrates improved accuracy and interpretability in detecting multiple anomaly types, offering practical benefits for real-time monitoring and decision support in manufacturing environments.

Abstract

In modern assembly pipelines, identifying anomalies is crucial in ensuring product quality and operational efficiency. Conventional single-modality methods fail to capture the intricate relationships required for precise anomaly prediction in complex predictive environments with abundant data and multiple modalities. This paper proposes a neurosymbolic AI and fusion-based approach for multimodal anomaly prediction in assembly pipelines. We introduce a time series and image-based fusion model that leverages decision-level fusion techniques. Our research builds upon three primary novel approaches in multimodal learning: time series and image-based decision-level fusion modeling, transfer learning for fusion, and knowledge-infused learning. We evaluate the novel method using our derived and publicly available multimodal dataset and conduct comprehensive ablation studies to assess the impact of our preprocessing techniques and fusion model compared to traditional baselines. The results demonstrate that a neurosymbolic AI-based fusion approach that uses transfer learning can effectively harness the complementary strengths of time series and image data, offering a robust and interpretable approach for anomaly prediction in assembly pipelines with enhanced performance. \noindent The datasets, codes to reproduce the results, supplementary materials, and demo are available at https://github.com/ChathurangiShyalika/NSF-MAP.

Figures (7)

• Figure 1: Architecture of NSF-MAP: Integration of time series and images for anomaly prediction, involving preprocessing, feature extraction using a pretrained EfficientNet (PC), and fusion with time series autoencoder outputs(TSA). The fusion model, enhanced by external process ontology knowledge, predicts the next time series output and classifies anomaly types.
• Figure 2: Experimental Results of Predicting Different Anomaly Types by NSF-MAP
• Figure 3: Performance of the Proposed Approach and Baselines with Varied Training and Testing Splits
• Figure 4: A Rocket Assembled by the Future Factories Lab. Any missing part is considered an anomaly: for example, the absence of Rocket body 1 is labeled as "NoBody1," while the absence of both Rocket body 1 and body 2 is labeled as "NoBody2, NoBody1."
• Figure 5: Future Factories Setup at Future Factories Lab. R01-Robot 1, R02-Robot 2, R03-Robot 3, R04-Robot 4