Leveraging Foundation Models To learn the shape of semi-fluid deformable objects

TL;DR

The subject of characterizing weld pool to define stable features that serve as information for further motion control objectives is addressed by employing different pipelines, which consists of characterizing fluid deformable objects through the use of a generative model that is trained using a teacher-student framework.

Abstract

One of the difficulties imposed on the manipulation of deformable objects is their characterization and the detection of representative keypoints for the purpose of manipulation. A keen interest was manifested by researchers in the last decade to characterize and manipulate deformable objects of non-fluid nature, such as clothes and ropes. Even though several propositions were made in the regard of object characterization, however researchers were always confronted with the need of pixel-level information of the object through images to extract relevant information. This usually is accomplished by means of segmentation networks trained on manually labeled data for this purpose. In this paper, we address the subject of characterizing weld pool to define stable features that serve as information for further motion control objectives. We achieve this by employing different pipelines. The first one consists of characterizing fluid deformable objects through the use of a generative model that is trained using a teacher-student framework. And in the second one we leverage foundation models by using them as teachers to characterize the object in the image, without the need of any pre-training and any dataset. The performance of knowledge distillation from foundation models into a smaller generative model shows prominent results in the characterization of deformable objects. The student network was capable of learning to retrieve the keypoitns of the object with an error of 13.4 pixels. And the teacher was evaluated based on its capacities to retrieve pixel level information represented by the object mask, with a mean Intersection Over Union (mIoU) of 75.26%.

Figures (6)

• Figure 2: Teacher-Student framework adopted for training a student network to characterize fluid deformable objects.
• Figure 3: Framework of the proposed DINO-SAM method
• Figure 4: The phases of finding the Gaussian heatmap for an image. (a) Initial image. (b) Output of DINO-SAM. (c) potential candidates clustered into 4 clusters. (d) The Gaussian heatmap at the center of each cluster. (e) Output of teacher : mask and heatmap.
• Figure 5: The setup used for dataset generation (arc-off)
• Figure 6: Examples of the different shapes encountered in the dataset. In red are the features that should be used to control the robot