Numerical Stability of DeepGOPlus Inference

TL;DR

The paper investigates the numerical stability of DeepGOPlus inference and the viability of reduced-precision inference using Monte Carlo Arithmetic (MCA) and VPREC. It documents that the DeepGOPlus CNN is numerically stable during inference, with negligible output perturbations under MCA (e.g., class-probability std dev around $10^{-8}$ and metric variability near $10^{-6}$), and that precision can be selectively reduced in inbound mode without harming performance, though outbound reductions often degrade results. The authors compare MCA tools Verificarlo and Verrou, quantify significant digits across samples, and identify TensorFloat-32 (TF32) as a hardware-supported option for single-precision reduction on Nvidia A100 GPUs. Overall, the work provides practical guidance for deploying DeepGOPlus across varied environments while highlighting hardware- and format-dependent constraints for reduced-precision inference.

Abstract

Convolutional neural networks (CNNs) are currently among the most widely-used deep neural network (DNN) architectures available and achieve state-of-the-art performance for many problems. Originally applied to computer vision tasks, CNNs work well with any data with a spatial relationship, besides images, and have been applied to different fields. However, recent works have highlighted numerical stability challenges in DNNs, which also relates to their known sensitivity to noise injection. These challenges can jeopardise their performance and reliability. This paper investigates DeepGOPlus, a CNN that predicts protein function. DeepGOPlus has achieved state-of-the-art performance and can successfully take advantage and annotate the abounding protein sequences emerging in proteomics. We determine the numerical stability of the model's inference stage by quantifying the numerical uncertainty resulting from perturbations of the underlying floating-point data. In addition, we explore the opportunity to use reduced-precision floating point formats for DeepGOPlus inference, to reduce memory consumption and latency. This is achieved by instrumenting DeepGOPlus' execution using Monte Carlo Arithmetic, a technique that experimentally quantifies floating point operation errors and VPREC, a tool that emulates results with customizable floating point precision formats. Focus is placed on the inference stage as it is the primary deliverable of the DeepGOPlus model, widely applicable across different environments. All in all, our results show that although the DeepGOPlus CNN is very stable numerically, it can only be selectively implemented with lower-precision floating-point formats. We conclude that predictions obtained from the pre-trained DeepGOPlus model are very reliable numerically, and use existing floating-point formats efficiently.

Figures (8)

• Figure 1: Illustration of Proposed Perturbation and Simulation of CNN Inference Methods
• Figure 2: Architecture of DeepGOPlus CNN (extracted from kulmanov2020deepgoplus)
• Figure 3: Variability Across Protein Function Class Probabilities. Standard deviations and significant digits were computed on 10 MCA samples, for each of 3,874 proteins.
• Figure 4: Difference in $AUPR$ Performance between Reduced Precision Formats and IEEE for GO Classes Using Inbound and Outbound Precision Only. The red X signifies the model crashed at these precision values. As all metrics experienced similar effects only $AUPR$ is shown, while the other metric heatmaps can be found in Figs. \ref{['fig_6']} and \ref{['fig_7']} of Appendix \ref{['appendix_a']}.
• Figure 5: Search for Optimal Precision and Exponent Values Before Performance Drop Off for VPREC Inbound Mode in Single-Precision. Double-precision is untouched and the green shaded region marks the area of acceptable values for reduced precision and comparable performance. Fig. \ref{['fig_8']} in Appendix \ref{['appendix_a']} demonstrates these findings hold when double-precision is reduced.