Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALBERTA: ALgorithm-Based Error Resilience in Transformer Architectures

Haoxuan Liu, Vasu Singh, Michał Filipiuk, Siva Kumar Sastry Hari

TL;DR

ALBERTA tackles the resilience of vision transformers to transient hardware faults by first mapping per-layer vulnerability and fault propagation, revealing that the prediction head is highly error-prone while larger models exhibit stronger resilience. It then introduces a low-overhead, checksum-based protection focused on the most vulnerable GEMM layers, complemented by a confidence-based epsilon model to distinguish numerical noise from true faults and a replay-based correction to maintain throughput. The approach yields over 99% coverage for misclassification-inducing faults with computation overhead below 0.2% and memory overhead below 0.01%, plus an average correction cost under 2%. Evaluations on DeiT-base and DeiT-tiny across FP16/32/64 on NVIDIA GPUs demonstrate practical viability for safety-critical vision tasks on modern hardware.

Abstract

Vision Transformers are being increasingly deployed in safety-critical applications that demand high reliability. It is crucial to ensure the correctness of their execution in spite of potential errors such as transient hardware errors. We propose a novel algorithm-based resilience framework called ALBERTA that allows us to perform end-to-end resilience analysis and protection of transformer-based architectures. First, our work develops an efficient process of computing and ranking the resilience of transformers layers. We find that due to the large size of transformer models, applying traditional network redundancy to a subset of the most vulnerable layers provides high error coverage albeit with impractically high overhead. We address this shortcoming by providing a software-directed, checksum-based error detection technique aimed at protecting the most vulnerable general matrix multiply (GEMM) layers in the transformer models that use either floating-point or integer arithmetic. Results show that our approach achieves over 99% coverage for errors that result in a mismatch with less than 0.2% and 0.01% computation and memory overheads, respectively. Lastly, we present the applicability of our framework in various modern GPU architectures under different numerical precisions. We introduce an efficient self-correction mechanism for resolving erroneous detection with an average of less than 2% overhead per error.

ALBERTA: ALgorithm-Based Error Resilience in Transformer Architectures

TL;DR

ALBERTA tackles the resilience of vision transformers to transient hardware faults by first mapping per-layer vulnerability and fault propagation, revealing that the prediction head is highly error-prone while larger models exhibit stronger resilience. It then introduces a low-overhead, checksum-based protection focused on the most vulnerable GEMM layers, complemented by a confidence-based epsilon model to distinguish numerical noise from true faults and a replay-based correction to maintain throughput. The approach yields over 99% coverage for misclassification-inducing faults with computation overhead below 0.2% and memory overhead below 0.01%, plus an average correction cost under 2%. Evaluations on DeiT-base and DeiT-tiny across FP16/32/64 on NVIDIA GPUs demonstrate practical viability for safety-critical vision tasks on modern hardware.

Abstract

Vision Transformers are being increasingly deployed in safety-critical applications that demand high reliability. It is crucial to ensure the correctness of their execution in spite of potential errors such as transient hardware errors. We propose a novel algorithm-based resilience framework called ALBERTA that allows us to perform end-to-end resilience analysis and protection of transformer-based architectures. First, our work develops an efficient process of computing and ranking the resilience of transformers layers. We find that due to the large size of transformer models, applying traditional network redundancy to a subset of the most vulnerable layers provides high error coverage albeit with impractically high overhead. We address this shortcoming by providing a software-directed, checksum-based error detection technique aimed at protecting the most vulnerable general matrix multiply (GEMM) layers in the transformer models that use either floating-point or integer arithmetic. Results show that our approach achieves over 99% coverage for errors that result in a mismatch with less than 0.2% and 0.01% computation and memory overheads, respectively. Lastly, we present the applicability of our framework in various modern GPU architectures under different numerical precisions. We introduce an efficient self-correction mechanism for resolving erroneous detection with an average of less than 2% overhead per error.
Paper Structure (18 sections, 13 figures)

This paper contains 18 sections, 13 figures.

Table of Contents

  1. Introduction
  2. Problem Statement
  3. Background
  4. Related Work
  5. Research Questions
  6. The ALBERTA Framework
  7. Vulnerability Analysis
  8. Resilience Analysis
  9. Resilience Implementation
  10. Execution and Evaluation
  11. Correction
  12. Results
  13. Evaluation Methodology
  14. Vulnerability Analysis
  15. Protecting Vision Transformers.
  16. ...and 3 more sections

Figures (13)

  • Figure 1: Overview of the self-attention module. Source: transformer-umich
  • Figure 2: Pipeline Overview for ALBERTA vulnerability analysis
  • Figure 3: Pipeline Overview for ALBERTA's resilience implementation
  • Figure 4: The error detection checksum compares the value of step-3 against the green vector in step-c element-wise for any numerical discrepancy and computes the batch average difference for distribution modeling during eps search.
  • Figure 5: $V_{orig}[i]$ computed as the fraction of MAC operations used in the $i$-th layer over the total number used by the network. The x-axis shows the layer number and the y-axis shows the $V_{orig}$ value.
  • ...and 8 more figures