Reflective-Net: Learning from Explanations

TL;DR

Reflective-Net introduces a self-reflective learning paradigm that uses explanations from explainability methods to augment training data and perform a brief second, deliberative pass during inference. By extending GradCAM to intermediate layers and concatenating multi-channel explanations with inputs, the method enables learning from multiple hypothetical outcomes (correct, predicted, random) and from different explanation sources. Empirical results across CIFAR, SVHN, FashionMNIST, and TinyImageNet show improved accuracy and faster early convergence compared to non-reflective baselines, with the strongest gains when explanations for the correct class are used at test time and when training leverages multiple explanations. The work highlights both the potential and the computational trade-offs of explanation-based self-reflection, and it opens avenues for data augmentation and alternative integration mechanisms like attention-based designs.

Abstract

We examine whether data generated by explanation techniques, which promote a process of self-reflection, can improve classifier performance. Our work is based on the idea that humans have the ability to make quick, intuitive decisions as well as to reflect on their own thinking and learn from explanations. To the best of our knowledge, this is the first time that the potential of mimicking this process by using explanations generated by explainability methods has been explored. We found that combining explanations with traditional labeled data leads to significant improvements in classification accuracy and training efficiency across multiple image classification datasets and convolutional neural network architectures. It is worth noting that during training, we not only used explanations for the correct or predicted class, but also for other classes. This serves multiple purposes, including allowing for reflection on potential outcomes and enriching the data through augmentation.

Figures (7)

• Figure 1: Reflective-Net: After a first classification (Step 1-2), a possible decision for the input is explained using a backward pass up to a specific layer (Step 3). Then, the explanation and input (Step 4) are used to get a second classification.
• Figure 2: Reflective-Net: Learning from explanations of correct and incorrect predictions. Predicted classes have grey boundaries. Reflective thinking based on either incorrect (left panel) or correct predictions (right panel) leads to better outcomes.
• Figure 3: Reflective-Net illustrated for VGG; A VGG network (grey boxes) is extended to a network using explanations. Explanations pass through two conv-layers (orange) and the output is appended; (each conv layer is followed by a batchnorm and a relu layer)
• Figure 4: Comparison of explanations using GradCAM and upsampled explanations of VGG on CIFAR-10 after the second conv layer ($L=Low$ in Table \ref{['fig:arch']})
• Figure 5: Explanation for correct class for an incorrect prediction. Blue shows an irrelevant feature for the correct class that still shows strong activations for the forward pass. Red shows a feature that should show strong activation according to the explanation but does not. Thus, an explanation of the correct class might provide more adequate information for classification.