Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image-based Deep Learning for Smart Digital Twins: a Review

Md Ruman Islam, Mahadevan Subramaniam, Pei-Chi Huang

TL;DR

This paper addresses the problem of designing and evaluating image-based Smart Digital Twins (SDTs) by surveying the associated data acquisition, preprocessing, and deep learning (DL) algorithms. It catalogs DL architectures—from CNN-based classifiers to R-CNN, YOLO, MediaPipe, Swin Transformer, and 3D variants—along with their suitability for DT tasks such as detection, classification, and real-time monitoring. The work provides a comparative performance perspective and documents recent applications across domains, while outlining challenges in data quality, interpretability, and real-time integration. It also offers future directions, including generative data augmentation, multimodal DL, and integration with 5G, edge computing, and IoT to enable robust, scalable image-based SDTs. Overall, the paper guides researchers and practitioners toward broader adoption of image-based SDTs and informs the development of new methods to replicate, predict, and optimize complex physical systems.

Abstract

Smart Digital twins (SDTs) are being increasingly used to virtually replicate and predict the behaviors of complex physical systems through continual data assimilation enabling the optimization of the performance of these systems by controlling the actions of systems. Recently, deep learning (DL) models have significantly enhanced the capabilities of SDTs, particularly for tasks such as predictive maintenance, anomaly detection, and optimization. In many domains, including medicine, engineering, and education, SDTs use image data (image-based SDTs) to observe and learn system behaviors and control their behaviors. This paper focuses on various approaches and associated challenges in developing image-based SDTs by continually assimilating image data from physical systems. The paper also discusses the challenges involved in designing and implementing DL models for SDTs, including data acquisition, processing, and interpretation. In addition, insights into the future directions and opportunities for developing new image-based DL approaches to develop robust SDTs are provided. This includes the potential for using generative models for data augmentation, developing multi-modal DL models, and exploring the integration of DL with other technologies, including 5G, edge computing, and IoT. In this paper, we describe the image-based SDTs, which enable broader adoption of the digital twin DT paradigms across a broad spectrum of areas and the development of new methods to improve the abilities of SDTs in replicating, predicting, and optimizing the behavior of complex systems.

Image-based Deep Learning for Smart Digital Twins: a Review

TL;DR

This paper addresses the problem of designing and evaluating image-based Smart Digital Twins (SDTs) by surveying the associated data acquisition, preprocessing, and deep learning (DL) algorithms. It catalogs DL architectures—from CNN-based classifiers to R-CNN, YOLO, MediaPipe, Swin Transformer, and 3D variants—along with their suitability for DT tasks such as detection, classification, and real-time monitoring. The work provides a comparative performance perspective and documents recent applications across domains, while outlining challenges in data quality, interpretability, and real-time integration. It also offers future directions, including generative data augmentation, multimodal DL, and integration with 5G, edge computing, and IoT to enable robust, scalable image-based SDTs. Overall, the paper guides researchers and practitioners toward broader adoption of image-based SDTs and informs the development of new methods to replicate, predict, and optimize complex physical systems.

Abstract

Smart Digital twins (SDTs) are being increasingly used to virtually replicate and predict the behaviors of complex physical systems through continual data assimilation enabling the optimization of the performance of these systems by controlling the actions of systems. Recently, deep learning (DL) models have significantly enhanced the capabilities of SDTs, particularly for tasks such as predictive maintenance, anomaly detection, and optimization. In many domains, including medicine, engineering, and education, SDTs use image data (image-based SDTs) to observe and learn system behaviors and control their behaviors. This paper focuses on various approaches and associated challenges in developing image-based SDTs by continually assimilating image data from physical systems. The paper also discusses the challenges involved in designing and implementing DL models for SDTs, including data acquisition, processing, and interpretation. In addition, insights into the future directions and opportunities for developing new image-based DL approaches to develop robust SDTs are provided. This includes the potential for using generative models for data augmentation, developing multi-modal DL models, and exploring the integration of DL with other technologies, including 5G, edge computing, and IoT. In this paper, we describe the image-based SDTs, which enable broader adoption of the digital twin DT paradigms across a broad spectrum of areas and the development of new methods to improve the abilities of SDTs in replicating, predicting, and optimizing the behavior of complex systems.
Paper Structure (20 sections, 2 figures, 3 tables)

This paper contains 20 sections, 2 figures, 3 tables.

Table of Contents

  1. introduction
  2. Architecture for Smart Digital Twins
  3. Images Data Acquisition and Preprocessing
  4. Sensors and Image Data Collection Techniques
  5. Data Preprocessing Methods for Image-based Digital Twins
  6. Challenges and Considerations in Handling Image Data for Digital Twins
  7. Image-based Intelligence Algorithms in Digital Twins
  8. CNN
  9. R-CNN Based Models
  10. YOLO Based Models
  11. MediaPipe
  12. Swin Transformer (Swin-T)
  13. 3D-VGG and 3D-ResNet
  14. Single Shot Detector (SSD)
  15. Performance Comparison of Deep Learning Models
  16. ...and 5 more sections

Figures (2)

  • Figure 1: A general architecture of the smart (DT) system.
  • Figure 2: Dataflow and processing in the physical and digital systems.