An Experimental Study of Semantic Continuity for Deep Learning Models

TL;DR

The paper addresses semantic discontinuity, where small semantic perturbations cause large output changes in deep models, undermining robustness and interpretability. It introduces a semantic continuity constraint that enforces smooth gradients by minimizing the semantic-change metric $DS(x,x')$ between original inputs $x$ and non-semantic perturbations $x'$, via the loss $Loss = Loss + \alpha Loss_{continuity}$ with $Loss_{continuity} = DS(x,x')$ and $x' = P(x)$. Empirical validation on ImageNet and CIFAR-100 with ResNet variants shows reduced semantic-discontinuity (lower $DS$), improved adversarial robustness, clearer and more focal explanations (IG, GradCAM, LIME), better transferability, and reduced bias on Color MNIST. The work highlights that aligning models with semantic neighborhoods yields tangible gains across trustworthiness metrics and offers a practical pathway to more human-aligned perception in deep learning systems.

Abstract

Deep learning models suffer from the problem of semantic discontinuity: small perturbations in the input space tend to cause semantic-level interference to the model output. We argue that the semantic discontinuity results from these inappropriate training targets and contributes to notorious issues such as adversarial robustness, interpretability, etc. We first conduct data analysis to provide evidence of semantic discontinuity in existing deep learning models, and then design a simple semantic continuity constraint which theoretically enables models to obtain smooth gradients and learn semantic-oriented features. Qualitative and quantitative experiments prove that semantically continuous models successfully reduce the use of non-semantic information, which further contributes to the improvement in adversarial robustness, interpretability, model transfer, and machine bias.

Figures (12)

• Figure 1: The first column shows the original sample $x_{A}$. The second column displays the data-augmented sample $x_{A'}$, which has been slightly adjusted for brightness, contrast, saturation, and hue. The third column presents the image $x_{B}$ from the same category. Notably, in the commonly used ResNet-50 pre-trained model, the representation distance between $x_{A}$ and $x_{A'}$ is greater than that between $x_{A}$ and $x_{B}$.
• Figure 2: Two-dimensional visualization of the representation of all samples in a specific category of the ImageNet Validation set. Black dots represent randomly selected images, while red dots indicate the representation positions of data-augmented images derived from one of the selected images. The left figure illustrates the result obtained using PCA, the middle one displays the TSNE result, and the right one showcases the outcome of UMAP.
• Figure 3: We aim to build constraints based on non-semantic transforms, guiding the model to focus on semantic information while completing tasks, thereby improving the model's trustworthiness metrics. In the figure, the left side shows the feature space of standard training, and the right side shows the feature space after applying the semantic continuity constraint. For a cat-dog classification task, we use a desert-forest analogy to illustrate the desired effect we want the model to achieve. In the original training mode, the model only focused on the classification accuracy. We hope that under semantic constraints, the model can also pay attention to semantic information at the same time. For example, the semantic information related to the desert forest in the example is also correctly transferred.
• Figure 4: We measured the semantic continuity of the four models under non-semantic perturbations, examining the effects of Brightness, Contrast, Saturation, and Hue transformations. The results show that the semantic continuity of the models significantly improved after adding the semantic continuity constraint.
• Figure 5: The semantic continuity and classification accuracy under different settings of $\alpha$ are shown in the figures. The left figure illustrates the accuracy of models with varying $\alpha$ values, while the right figure displays the semantic continuity of models with different $\alpha$ values.