Two out of Three (ToT): using self-consistency to make robust predictions

TL;DR

Two out of Three (ToT) addresses the opacity of deep learning decisions by introducing self-consistency through multiple internal viewpoints. It generates two additional predictions from input views: ROI-based second predictions $P_{2nd}$ and a third decision $P_{3rd}$ derived from hidden features, then applies a two-out-of-three consistency rule to decide or abstain, with Gaussian blur on ROIs enhancing uncertainty detection. Across ImageNet-derived subsets Mixed_13 and Geirhos_16 and five architectures (ResNets, DenseNet, VGG, ViT), ToT improves high-confidence accuracy, enables abstention on a subset of inputs, and demonstrates robust performance against PGD and AutoAttack adversaries, achieving final non-null accuracies around $60$–$90\%$ depending on model. This approach offers a practical pathway to safer DL in high-stakes applications by leveraging self-consistency and multi-view perspectives within a single model, potentially reducing critical errors prior to deployment.

Abstract

Deep learning (DL) can automatically construct intelligent agents, deep neural networks (alternatively, DL models), that can outperform humans in certain tasks. However, the operating principles of DL remain poorly understood, making its decisions incomprehensible. As a result, it poses a great risk to deploy DL in high-stakes domains in which mistakes or errors may lead to critical consequences. Here, we aim to develop an algorithm that can help DL models make more robust decisions by allowing them to abstain from answering when they are uncertain. Our algorithm, named `Two out of Three (ToT)', is inspired by the sensitivity of the human brain to conflicting information. ToT creates two alternative predictions in addition to the original model prediction and uses the alternative predictions to decide whether it should provide an answer or not.

Figures (9)

• Figure 1: Overview of ToT. (A), 4 predictions used in ToT. In principle, ToT uses original, second and third predictions, but we find it is beneficial to use additional second predictions without blurring the images. Consequently, we have $P_{orig}$, $P_{2nd}$, $P_{3rd}$ and $P_{2nd'}$. (B), Comparing $P_{orig}$ and $P_{2nd}$. (C), Determine a final answer, which can be either 'Null' or one of the top-2 predictions from symbols $\vec{S}$.
• Figure 1: Evaluation of ToT on adversarial inputs crafted by AutoAttack attack. (A)-(E), evaluation of ResNet18, ResNet50, VGG19, DenseNet121 and ViT, respectively. We estimate failure ratio (FR) of detection, the accuracy of final answers (ACC), the rate of inaccurate final answers (ACIC) and the rate of inputs(ACUC), on which ToT refuses to make prediction. They are shown in blue, orange, green and red, respectively. We show evaluation on Mixed_13 in the left column and those on Geirhos_16 in the right column.
• Figure 2: (A), Accuracy of high confident predictions on Mixed_13. The blue bars denote the accuracy of high confident predictions, whereas the orange bars denote the accuracy of original predictions. (B), Rate of low confident predictions (i.e., the number of inputs, which original and second predictions disagree). (C), Accuracy of high confident predictions depending on the kernel size ($\sigma$) of Gaussian blur kernel. (D), Rate of low confident predictions depending on the kernel size ($\sigma$) of Gaussian blur kernel. R50, V19, D121 and VIT denote ResNet18, ResNet50, VGG19, DenseNet121 and Vision Transformer.
• Figure 2: Evaluation of ToT on normal inputs. (A)-(E), evaluation of ResNet18, ResNet50, VGG19, DenseNet121 and ViT, respectively. We show evaluation on Mixed_13 in the left column and those on Geirhos_16 in the right column. See the text for details.
• Figure 3: (A), Accuracy of high confident predictions on Geirhos_16. The blue bars denote the accuracy of high confident predictions, whereas the orange bars denote the accuracy of original predictions. (B), Rate of low confident predictions (i.e., the number of inputs, which original and second predictions disagree). (C), Accuracy of high confident predictions depending on the kernel size ($\sigma$) of Gaussian blur kernel. (D), Rate of low confident predictions depending on the kernel size ($\sigma$) of Gaussian blur kernel.