PolyLUT: Ultra-low Latency Polynomial Inference with Hardware-Aware Structured Pruning

TL;DR

PolyLUT presents a DNN-FPGA co-design that trains neurons to compute multivariate polynomials up to degree $D$ and maps them into netlists of L-LUTs for ultra-low latency FPGA inference. By embedding polynomial evaluation inside LUTs and employing hardware-aware structured pruning, the approach achieves shallower networks with substantial latency and area reductions while maintaining accuracy across NID, MNIST, and jet substructure tagging. The method outperforms prior LUT-based architectures in latency, LUT count, and robustness to seed variability, and demonstrates concrete improvements (e.g., up to ~2–18x latency reductions) across multiple datasets. Future work includes scaling LUTs, exploring don't-care conditions, and introducing NAS to automate architecture selection.

Abstract

Standard deep neural network inference involves the computation of interleaved linear maps and nonlinear activation functions. Prior work for ultra-low latency implementations has hardcoded these operations inside FPGA lookup tables (LUTs). However, FPGA LUTs can implement a much greater variety of functions. In this paper, we propose a novel approach to training DNNs for FPGA deployment using multivariate polynomials as the basic building block. Our method takes advantage of the flexibility offered by the soft logic, hiding the polynomial evaluation inside the LUTs with minimal overhead. By using polynomial building blocks, we achieve the same accuracy using considerably fewer layers of soft logic than by using linear functions, leading to significant latency and area improvements. LUT-based implementations also face a significant challenge: the LUT size grows exponentially with the number of inputs. Prior work relies on a priori fixed sparsity, with results heavily dependent on seed selection. To address this, we propose a structured pruning strategy using a bespoke hardware-aware group regularizer that encourages a particular sparsity pattern that leads to a small number of inputs per neuron. We demonstrate the effectiveness of PolyLUT on three tasks: network intrusion detection, jet identification at the CERN Large Hadron Collider, and MNIST.

Figures (14)

• Figure 1: Structural use of LUTs for a single output channel in prior works (a,b) and our approach (c).
• Figure 2: Illustration of a toy three-layer neural network.
• Figure 3: Input transformations visualized as contour graphs at the output of each neuron. The black and red dotted spirals are the datapoints used for training.
• Figure 4: Three-stage network training pipeline.
• Figure 5: Structured network pruning.