Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hawkeye: Reproducing GPU-Level Non-Determinism

Erez Badash, Dan Boneh, Ilan Komargodski, Megha Srivastava

Abstract

We present Hawkeye, a system for analyzing and reproducing GPU-level arithmetic operations. Using our framework, anyone can re-execute on a CPU the exact matrix multiplication operations underlying a machine learning model training or inference workflow that was executed on an NVIDIA GPU, without any precision loss. This is in stark contrast to prior approaches to verifiable machine learning, which either introduce significant computation overhead to the original model owner, or suffer from non-robustness and quality degradation. The main technical contribution of Hawkeye is a systematic sequence of carefully crafted tests that study rounding direction, subnormal number handling, and order of (non-associative) accumulation during matrix multiplication on NVIDIA's Tensor Cores. We test and evaluate our framework on multiple NVIDIA GPU architectures ( Ampere, Hopper, and Lovelace) and precision types (FP16, BFP16, FP8). In all test cases, Hawkeye enables perfect reproduction of matrix multiplication on a CPU, paving the way for efficient and trustworthy third-party auditing of ML model training and inference.

Hawkeye: Reproducing GPU-Level Non-Determinism

Abstract

We present Hawkeye, a system for analyzing and reproducing GPU-level arithmetic operations. Using our framework, anyone can re-execute on a CPU the exact matrix multiplication operations underlying a machine learning model training or inference workflow that was executed on an NVIDIA GPU, without any precision loss. This is in stark contrast to prior approaches to verifiable machine learning, which either introduce significant computation overhead to the original model owner, or suffer from non-robustness and quality degradation. The main technical contribution of Hawkeye is a systematic sequence of carefully crafted tests that study rounding direction, subnormal number handling, and order of (non-associative) accumulation during matrix multiplication on NVIDIA's Tensor Cores. We test and evaluate our framework on multiple NVIDIA GPU architectures ( Ampere, Hopper, and Lovelace) and precision types (FP16, BFP16, FP8). In all test cases, Hawkeye enables perfect reproduction of matrix multiplication on a CPU, paving the way for efficient and trustworthy third-party auditing of ML model training and inference.
Paper Structure (35 sections, 12 equations, 2 figures, 1 algorithm)

This paper contains 35 sections, 12 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation.
  3. Our work.
  4. The Challenge: Reproducing NNs
  5. GPU architecture.
  6. Sources of non-determinism.
  7. Problem statement.
  8. Floating-Point Preliminaries
  9. Multiplication and Addition of FPs
  10. Non-associativity.
  11. Overview of Hawkeye
  12. Tensor Core Characterization Methodology.
  13. Target Tests
  14. Reproducing FP16 Workflow on Ampere
  15. Full-Precision FP16 Products Verification
  16. ...and 20 more sections

Figures (2)

  • Figure 1: Computational graph of the two-stage tensor core accumulation in a pyramid structure. The initial accumulator $C_{i,j}$ and the first 8 products are summed into an intermediate result, which is then summed with the last 8 products.
  • Figure 2: Computational graph of Hooper tensor core accumulation in a pyramid structure. The initial accumulator $C_{i,j}$ and the 16 products are summed into the final result