PerfCam: Digital Twinning for Production Lines Using 3D Gaussian Splatting and Vision Models

TL;DR

PerfCam addresses the need for low-cost digital twins of production lines by leveraging camera data and 3D Gaussian Splatting to reconstruct 3D scenes and extract KPIs directly from visual inputs. The approach integrates COLMAP-based SfM/MVS with SuGaR 3D Gaussian Splatting and YOLOv11-based object detection to create a live digital twin, augmented with sensory data and situated visualizations in Unity. Key contributions include: (i) an open-source PoC framework that delivers real-time 3D reconstruction, object detection/tracking, and KPI extraction; (ii) KPI computation including throughput, conveyor speed, and OEE, where OEE is $A \cdot P \cdot Q$; and (iii) an openly published dataset and a real-world deployment in a pharmaceutical-like test line. The results show accurate item localization and KPI tracking with modest error (e.g., mean OEE error around $2.5\%$), demonstrating the practical viability and potential to scale digital-twin analytics in smart manufacturing.

Abstract

We introduce PerfCam, an open source Proof-of-Concept (PoC) digital twinning framework that combines camera and sensory data with 3D Gaussian Splatting and computer vision models for digital twinning, object tracking, and Key Performance Indicators (KPIs) extraction in industrial production lines. By utilizing 3D reconstruction and Convolutional Neural Networks (CNNs), PerfCam offers a semi-automated approach to object tracking and spatial mapping, enabling digital twins that capture real-time KPIs such as availability, performance, Overall Equipment Effectiveness (OEE), and rate of conveyor belts in the production line. We validate the effectiveness of PerfCam through a practical deployment within realistic test production lines in the pharmaceutical industry and contribute an openly published dataset to support further research and development in the field. The results demonstrate PerfCam's ability to deliver actionable insights through its precise digital twin capabilities, underscoring its value as an effective tool for developing usable digital twins in smart manufacturing environments and extracting operational analytics.

Figures (10)

• Figure 1: Visual representations in PerfCam vs. ground truth. (a) A 3D reconstruction view based on 3D Gaussian Splatting with a GUI to see various KPIs, (b) a low-latency annotated view of the production line from different camera angles, and (c) a 2.5D map of the production line, which reflects the ground truth (d).
• Figure 2: System workflow of PerfCam. PerfCam uses 4-6 small robotic arms with RGB cameras to capture frames around the production line, while sensors record multimodal data in a time-series DB. It reconstructs a detailed 3D model using COLMAP and Gaussian Splatting and then employs a CNN-based object detection and tracking to extract KPIs such as conveyor belt throughput, number of stops, and OEE. Finally, PerfCam provides 3D reconstructed and annotated views, with annotated visualizations enabling users to monitor and improve production KPIs and processes interactively.
• Figure 3: Snapshots from video footage captured by Cameras 1–4 during the experiments on the test line (second row) alongside corresponding views in the PerfCam's digital twin (first row).
• Figure 4: Sensor measurements from the motor.
• Figure 5: Availability (in %) of conveyor belt calculated based on the changes in motor's acceleration.