Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Hidden Bloat in Machine Learning Systems

Huaifeng Zhang, Ahmed Ali-Eldin

TL;DR

The paper addresses the problem of software bloat in ML systems by introducing Negativa-ML, a tool that debloats both CPU and GPU code inside ML shared libraries. It proposes a two-component approach—a kernel detector that identifies used GPU kernels with low overhead and a kernel locator that locates and retains the necessary cubins and elements—followed by a compaction stage to remove the rest. Evaluated on four ML frameworks across ten workloads and 300+ shared libraries, Negativa-ML achieves up to 55% total file-size reduction, with GPU code reductions up to 75% and CPU code reductions up to 72%, and reports significant improvements in peak memory usage and startup time. The results demonstrate that GPU code is a major source of bloat in ML frameworks and that a small subset of libraries disproportionately contributes to reductions, suggesting practical benefits for deployment in resource-constrained environments and edge data centers.

Abstract

Software bloat refers to code and features that is not used by a software during runtime. For Machine Learning (ML) systems, bloat is a major contributor to their technical debt leading to decreased performance and resource wastage. In this work, we present, Negativa-ML, a novel tool to identify and remove bloat in ML frameworks by analyzing their shared libraries. Our approach includes novel techniques to detect and locate unnecessary code within device code - a key area overlooked by existing research, which focuses primarily on host code. We evaluate Negativa-ML using four popular ML frameworks across ten workloads over 300 shared libraries. The results demonstrate that the ML frameworks are highly bloated on both the device and host code side. On average, Negativa-ML reduces the device code size in these frameworks by up to 75% and the host code by up to 72%, resulting in total file size reductions of up to 55%. The device code is a primary source of bloat within ML frameworks. Through debloating, we achieve reductions in peak host memory usage, peak GPU memory usage, and execution time by up to 74.6%, 69.6%, and 44.6%, respectively.

The Hidden Bloat in Machine Learning Systems

TL;DR

The paper addresses the problem of software bloat in ML systems by introducing Negativa-ML, a tool that debloats both CPU and GPU code inside ML shared libraries. It proposes a two-component approach—a kernel detector that identifies used GPU kernels with low overhead and a kernel locator that locates and retains the necessary cubins and elements—followed by a compaction stage to remove the rest. Evaluated on four ML frameworks across ten workloads and 300+ shared libraries, Negativa-ML achieves up to 55% total file-size reduction, with GPU code reductions up to 75% and CPU code reductions up to 72%, and reports significant improvements in peak memory usage and startup time. The results demonstrate that GPU code is a major source of bloat in ML frameworks and that a small subset of libraries disproportionately contributes to reductions, suggesting practical benefits for deployment in resource-constrained environments and edge data centers.

Abstract

Software bloat refers to code and features that is not used by a software during runtime. For Machine Learning (ML) systems, bloat is a major contributor to their technical debt leading to decreased performance and resource wastage. In this work, we present, Negativa-ML, a novel tool to identify and remove bloat in ML frameworks by analyzing their shared libraries. Our approach includes novel techniques to detect and locate unnecessary code within device code - a key area overlooked by existing research, which focuses primarily on host code. We evaluate Negativa-ML using four popular ML frameworks across ten workloads over 300 shared libraries. The results demonstrate that the ML frameworks are highly bloated on both the device and host code side. On average, Negativa-ML reduces the device code size in these frameworks by up to 75% and the host code by up to 72%, resulting in total file size reductions of up to 55%. The device code is a primary source of bloat within ML frameworks. Through debloating, we achieve reductions in peak host memory usage, peak GPU memory usage, and execution time by up to 74.6%, 69.6%, and 44.6%, respectively.

Paper Structure

This paper contains 18 sections, 1 equation, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. ML Frameworks
  4. Software Bloat
  5. Software Debloating
  6. Methodology
  7. Kernel Detector
  8. Kernel Locator
  9. Experiments
  10. Overview of Bloat in ML Frameworks
  11. Shared Library Level Analysis
  12. Function and Element Level Analysis
  13. Runtime Performance Analysis
  14. Evaluation on Different GPUs
  15. Performance of Negativa-ML
  16. ...and 3 more sections

Figures (7)

  • Figure 1: Distribution of CPU code and GPU code in the top 4 largest shared libraries in PyTorch.
  • Figure 2: Overview of Negativa-ML. The components proposed in this work are highlighted in yellow rectangles.
  • Figure 3: Kernel detector workflow.
  • Figure 4: Structure of GPU code.
  • Figure 5: Violin plots of distribution of CPU and GPU code reduction in size and function count.
  • ...and 2 more figures