Automated Annotation of Shearographic Measurements Enabling Weakly Supervised Defect Detection

TL;DR

The paper tackles the lack of annotated shearography data by introducing an automated, zero-shot annotation pipeline that combines Grounded DINO for bounding boxes with SAM for segmentation, feeding into YOLOv8-based defect detectors. The approach enables weakly supervised training and scalable dataset creation, achieving strong binary classification and competitive localization compared to manual labels, though localization metrics are sensitive to bounding-box fragmentation. Prompt choice significantly impacts performance, and qualitative analysis highlights both the benefits of richer segmentation and the need for post-processing to mitigate duplicate or overlapping detections. The work points to future directions including exemplar prompting (SAM3), domain-adaptive fine-tuning, and multi-class defect characterization, with code and data to be released.

Abstract

Shearography is an interferometric technique sensitive to surface displacement gradients, providing high sensitivity for detecting subsurface defects in safety-critical components. A key limitation to industrial adoption is the lack of high-quality annotated datasets, since manual labeling remains labor-intensive, subjective, and difficult to standardize. We introduce an automated workflow that generates defect annotations from shearography measurements using deep learning, producing high-resolution segmentation and bounding-box labels. Evaluation against expert-labeled data demonstrates sufficient accuracy to enable weakly supervised training, reducing manual effort and supporting scalable dataset creation for robust defect detection.

Figures (4)

• Figure 1: Overview of the zero-shot annotation pipeline. Bounding boxes are generated by a pre-trained zero-shot detector (a), refined into segmentation masks by a zero-shot segmentor (b), and subsequently used for supervised training of defect detection models (c).
• Figure 2: ROC and PR curves for all trained YOLOv8 models using bounding boxes. Since the models achieve very high overall performance, only a selected portion of the curves is displayed to clearly illustrate the differences between the models.
• Figure 3: Bounding boxes and confidence scores predicted by Grounded DINO using the “Two Circles” prompt and corresponding SAM segmentation masks. (a) Example of correctly filtered detections. (b) Excessive bounding boxes around a streak-like artifact. (c) Duplicate bounding boxes for left single defect. (d) Same region at a later time, with right defect split into multiple boxes.
• Figure 4: Validation results of the YOLO segmentation model trained on masks generated by the zero-shot Grounded-DINO + SAM pipeline using the “Two Circles“ prompt. Segmentation masks are blue, details are highlighted with orange boxes. For each case (a–f), the upper image shows the automatically generated segmentation mask, and the lower image shows the corresponding YOLO prediction. (a–c) illustrate the close correspondence between automatic labels and model output. (d) shows a defect region missed by the zero-shot pipeline but successfully detected by YOLO (detail scale 2:1); (e) shows a case where YOLO predicts a smaller, fragmented mask that no longer captures the full defect; and (f) demonstrates how an incorrectly placed automatic label is propagated through the trained model.