Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Panther: Faster and Cheaper Computations with Randomized Numerical Linear Algebra

Fahd Seddik, Abdulrahman Elbedewy, Gaser Sami, Mohamed Abdelmoniem, Yahia Zakaria

TL;DR

Panther tackles memory and compute bottlenecks in training large neural networks by integrating randomized numerical linear algebra (RandNLA) techniques into PyTorch through a production-grade, drop-in library. It unifies core RandNLA primitives within a three-layer stack backed by a high-performance pawX C++/CUDA backend and an Optuna-based AutoTuner to automatically select configurations under resource constraints. Empirical results show substantial gains, including up to 75 percent memory reduction on BERT with competitive MLM loss, and modest accuracy changes when compressing models like ResNet-50, demonstrating practical viability for large-scale models. Overall, Panther lowers the barrier to adopting RandNLA in real ML workflows, enabling faster experimentation and deployment with minimal engineering effort.

Abstract

Training modern deep learning models is increasingly constrained by GPU memory and compute limits. While Randomized Numerical Linear Algebra (RandNLA) offers proven techniques to compress these models, the lack of a unified, production-grade library prevents widely adopting these methods. We present Panther, a PyTorch-compatible library that consolidates established RandNLA algorithms into a single high-performance framework. Panther engineers efficient, drop-in replacements for standard components including sketched linear layers, 2D convolution, multi-head attention, and randomized matrix decompositions (such as pivoted CholeskyQR). By implementing a custom C++/CUDA backend (pawX), Panther provides an optimized implementation that can run on both CPUs and GPUs. We demonstrate the effectiveness of RandNLA techniques and Panther's ease of adoption. By replacing standard PyTorch linear layers with Panther layers (requiring only a few lines of code) we achieve significant memory savings (up to 75%) on BERT while maintaining comparable loss. Source code is available (MIT License) at https://github.com/FahdSeddik/panther, along with demonstration video at https://youtu.be/7M3RQb4KWxs.

Panther: Faster and Cheaper Computations with Randomized Numerical Linear Algebra

TL;DR

Panther tackles memory and compute bottlenecks in training large neural networks by integrating randomized numerical linear algebra (RandNLA) techniques into PyTorch through a production-grade, drop-in library. It unifies core RandNLA primitives within a three-layer stack backed by a high-performance pawX C++/CUDA backend and an Optuna-based AutoTuner to automatically select configurations under resource constraints. Empirical results show substantial gains, including up to 75 percent memory reduction on BERT with competitive MLM loss, and modest accuracy changes when compressing models like ResNet-50, demonstrating practical viability for large-scale models. Overall, Panther lowers the barrier to adopting RandNLA in real ML workflows, enabling faster experimentation and deployment with minimal engineering effort.

Abstract

Training modern deep learning models is increasingly constrained by GPU memory and compute limits. While Randomized Numerical Linear Algebra (RandNLA) offers proven techniques to compress these models, the lack of a unified, production-grade library prevents widely adopting these methods. We present Panther, a PyTorch-compatible library that consolidates established RandNLA algorithms into a single high-performance framework. Panther engineers efficient, drop-in replacements for standard components including sketched linear layers, 2D convolution, multi-head attention, and randomized matrix decompositions (such as pivoted CholeskyQR). By implementing a custom C++/CUDA backend (pawX), Panther provides an optimized implementation that can run on both CPUs and GPUs. We demonstrate the effectiveness of RandNLA techniques and Panther's ease of adoption. By replacing standard PyTorch linear layers with Panther layers (requiring only a few lines of code) we achieve significant memory savings (up to 75%) on BERT while maintaining comparable loss. Source code is available (MIT License) at https://github.com/FahdSeddik/panther, along with demonstration video at https://youtu.be/7M3RQb4KWxs.
Paper Structure (12 sections, 3 figures)

This paper contains 12 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Panther Design
  3. Architecture and Core Engine
  4. AutoTuner Module
  5. Tool Usage
  6. During Development use-case
  7. After Development use-case
  8. Evaluation
  9. Runtime and Memory
  10. Quality
  11. Related Work
  12. Conclusion

Figures (3)

  • Figure 1: Forward pass runtime (ms) for the sketched Linear layer kasiviswanathan2017deepneuralnetworkapproximation compared to PyTorch. Run is for input and output features of 8192 and varies the introduced hyperparameters number of terms ($l$) and low rank ($k$)
  • Figure 2: Forward pass runtime (ms) for the sketched Conv2D layer kasiviswanathan2017deepneuralnetworkapproximation compared to PyTorch. Run is for input and output channels of $256 \times 2048$ with a squared kernel and image of size $9$ and $64$ respectively. We vary the introduced hyperparameters number of terms ($l$) and low rank ($k$)
  • Figure 3: Forward pass memory (MB) comparison for Panther's RandMultiHeadAttention using linear attention of Performers choromanski2022rethinkingattentionperformers compared to PyTorch. Run is for embed dimension of 512 using a softmax kernel and varies sequence length, number of heads, and the introduced random features hyperparameter.