A graph generation pipeline for critical infrastructures based on heuristics, images and depth data

TL;DR

The paper tackles the high-cost barrier of generating faithful digital twins for critical infrastructure by proposing a photogrammetry-based graph-generation pipeline that uses RGB images, depth data, and camera poses. It combines YOLOv8-based object detection with heuristic, user-defined rules to infer object relations and build a relational graph suitable for simulations and decision-making. The authors validate their approach on two synthetic hydraulic systems created in Unreal Engine 5, achieving graphs close to ground truth while highlighting practical limitations and the method's flexibility. This cost-effective, interpretable framework aims to broaden access to digital twins and support high-stakes infrastructure planning and resilience analyses.

Abstract

Virtual representations of physical critical infrastructures, such as water or energy plants, are used for simulations and digital twins to ensure resilience and continuity of their services. These models usually require 3D point clouds from laser scanners that are expensive to acquire and require specialist knowledge to use. In this article, we present a graph generation pipeline based on photogrammetry. The pipeline detects relevant objects and predicts their relation using RGB images and depth data generated by a stereo camera. This more cost-effective approach uses deep learning for object detection and instance segmentation of the objects, and employs user-defined heuristics or rules to infer their relations. Results of two hydraulic systems show that this strategy can produce graphs close to the ground truth while its flexibility allows the method to be tailored to specific applications and its transparency qualifies it to be used in the high stakes decision-making that is required for critical infrastructures.

Figures (12)

• Figure 1: Synthetic hydraulic systems. (A) shows system 1 consisting of pipes, one pump, one tank, one valve and one sprinkler. (B), (C) and (D) show system 2 consisting of pipes, two pumps, two tanks, three valves and four sprinklers.
• Figure 2: Flowchart of the graph generation pipeline. Pipe and non-pipe objects are treated differently and are combined in the 'Graph generation' module.
• Figure 3: Object detection of pumps, tanks and valves including one keypoint for tanks and two keypoints for pumps and valves. Bounding boxes indicate detected objects with class label and class confidence score. Keypoints are indicated by yellow dots.
• Figure 4: Object matching across individual images. (A) shows the matched objects of a pump, a tank and a valve and (B) shows the matched pipe objects. Each color indicates a distinct object.
• Figure 5: Instance segmentation of pipe objects. Bounding boxes indicate detected objects with class label and class confidence score, while segmented objects are shaded blue.