mBEST: Realtime Deformable Linear Object Detection Through Minimal Bending Energy Skeleton Pixel Traversals

Abstract

Robotic manipulation of deformable materials is a challenging task that often requires realtime visual feedback. This is especially true for deformable linear objects (DLOs) or "rods", whose slender and flexible structures make proper tracking and detection nontrivial. To address this challenge, we present mBEST, a robust algorithm for the realtime detection of DLOs that is capable of producing an ordered pixel sequence of each DLO's centerline along with segmentation masks. Our algorithm obtains a binary mask of the DLOs and then thins it to produce a skeleton pixel representation. After refining the skeleton to ensure topological correctness, the pixels are traversed to generate paths along each unique DLO. At the core of our algorithm, we postulate that intersections can be robustly handled by choosing the combination of paths that minimizes the cumulative bending energy of the DLO(s). We show that this simple and intuitive formulation outperforms the state-of-the-art methods for detecting DLOs with large numbers of sporadic crossings ranging from curvatures with high variance to nearly-parallel configurations. Furthermore, our method achieves a significant performance improvement of approximately 50% faster runtime and better scaling over the state of the art.

Figures (5)

• Figure 1: Overview of the mBEST processing pipeline. An input image (a) is converted to a binary mask (b) using a segmentation method. The binary mask is then converted to a skeleton pixel representation (c), where the connectivity and centerlines of the DLOs are preserved as a single-pixel wide structure and keypoints, such as intersections and ends, are detected. This is followed by a series of refinement steps to maintain the topological correctness of the skeleton: split ends (d1) are pruned (d2) and pixels representing a single topological intersection (e1) are clustered, matched, and replaced with a more intuitive intersection (e2). Finally, the DLOs are delineated (f) by traversing skeleton pixels and choosing minimal cumulative bending energy paths.
• Figure 2: Examples of split ends that may occur during the skeletonization process. Row (a) shows split ends that may occur at an actual topological end, while Row (b) shows a split end along a segment produced by a jagged mask. For both examples, the first column shows the binary mask; the second shows the split end after skeletonization, and the third shows the topologically correct structure after pruning.
• Figure 3: The intersection clustering, matching, replacement, and optimal path generation pipeline. Two sample intersections are shown where skeletonization results in a 2Y-shaped crossing (a1) and an X-shaped crossing (a2). As 2Y-shaped crossings are topologically incorrect, we replace them by replacing the intersection pixels (b) in two stages: the first involves clustering adjacent pixels and the second involves pair matching nearby clusters. Using the centroid location of the matched clusters, we then replace the intersection (c) by creating new ends and having new segments sprout and connect to the centroid. Finally, (d) the new generated ends and segments are used to discover the combination of paths that minimizes the cumulative bending energy of the DLO.
• Figure 4: Sample segmentations for the simple configuration against complex background dataset. Each row shows segmentation results for a different image with the left column indicating the dataset to which the image belongs. Columns 2--5 show Ariadne+, FASTDLO, RT-DLO, and mBEST results, respectively. The right column shows the ground truth. Note the failure to properly handle intersections for all baseline algorithms, especially when strands are nearly parallel. In fact, RT-DLO can be seen to produce an unintuitive output for the last example where certain wires are labeled multiple times.
• Figure 5: Sample segmentations for the complex configuration against simple background dataset. Each row shows segmentation results for a different image with the left column indicating the dataset to which the image belongs. Columns 2--5 show results for Ariadne+, FASTDLO, RT-DLO, and mBEST, respectively. The right column shows the ground truth. Several cases of incorrect intersection handling can be observed for all the baseline algorithms, whereas mBEST robustly handles intersections using its simple bending energy optimization.