Sample-Centric Multi-Task Learning for Detection and Segmentation of Industrial Surface Defects

TL;DR

The paper tackles the misalignment between pixel-level segmentation metrics and production-level defect decisions in industrial inspection. It introduces a sample-centric multi-task learning framework built on a shared encoder, with a segmentation branch and a lightweight classifier head CaP, and formalizes evaluation with Sample_mIoU, Seg_Accuracy, and Seg_Recall. The approach jointly optimizes pixel-level localization and sample-level defect presence, using a joint loss $L=\lambda_{seg}L_{seg}+\lambda_{cls}L_{cls}$ with $\lambda_{seg}=\lambda_{cls}=0.5$, and demonstrates that this boosts small-defect recall and sample-level reliability while maintaining localization quality. Empirical results on KolektorSDD2 and Crack show substantial improvements in per-item decision stability and defect localization completeness, providing a practical path toward more reliable first-pass yield in production. The work also introduces a pluggable CaP module that enables easy integration with existing segmentation backbones, thereby facilitating industry adoption.

Abstract

Industrial surface defect inspection for sample-wise quality control (QC) must simultaneously decide whether a given sample contains defects and localize those defects spatially. In real production lines, extreme foreground-background imbalance, defect sparsity with a long-tailed scale distribution, and low contrast are common. As a result, pixel-centric training and evaluation are easily dominated by large homogeneous regions, making it difficult to drive models to attend to small or low-contrast defects-one of the main bottlenecks for deployment. Empirically, existing models achieve strong pixel-overlap metrics (e.g., mIoU) but exhibit insufficient stability at the sample level, especially for sparse or slender defects. The root cause is a mismatch between the optimization objective and the granularity of QC decisions. To address this, we propose a sample-centric multi-task learning framework and evaluation suite. Built on a shared-encoder architecture, the method jointly learns sample-level defect classification and pixel-level mask localization. Sample-level supervision modulates the feature distribution and, at the gradient level, continually boosts recall for small and low-contrast defects, while the segmentation branch preserves boundary and shape details to enhance per-sample decision stability and reduce misses. For evaluation, we propose decision-linked metrics, Seg_mIoU and Seg_Recall, which remove the bias of classical mIoU caused by empty or true-negative samples and tightly couple localization quality with sample-level decisions. Experiments on two benchmark datasets demonstrate that our approach substantially improves the reliability of sample-level decisions and the completeness of defect localization.

Figures (7)

• Figure 1: High Scores $\neq$ Good Models: The Disconnect Between Evaluation Metrics and Real-World Needs
• Figure 2: MTL Network: Joint Segmentation–Classification Learning on a Shared Encoder
• Figure 3: Abstract Structure and Information Flow of the CaP Plugin.
• Figure 4: Segmentation Performance Comparison of Fully Supervised Models on the KSDD2 Dataset
• Figure 5: Segmentation Performance Comparison of Fully Supervised Models on the Crack Dataset