N-TORC: Native Tensor Optimizer for Real-time Constraints

TL;DR

N-TORC introduces a two-stage optimization framework for real-time FPGA inference: a neural-architecture search that balances accuracy and workload, followed by a MIP-based deployment step that assigns per-layer reuse factors to meet strict latency while minimizing resource usage. The approach relies on data-driven performance and cost models trained specifically for HLS4ML layer types (conv1d, LSTM, dense) and integrates these into a linearized optimization problem solved by a MIP solver, yielding Pareto-optimal networks and deployment configurations orders of magnitude faster than exhaustive stochastic search. On the DROPBEAR benchmark, N-TORC achieves higher accuracy than prior efforts and delivers end-to-end latency close to 200 μs with throughput from 11 to 39 Mops/s while using modest FPGA resources, demonstrating practical impact for high-rate, deterministic control tasks. The work also identifies limitations, notably the absence of quantization optimization, which the authors plan to integrate to further reduce resource costs.

Abstract

Compared to overlay-based tensor architectures like VTA or Gemmini, compilers that directly translate machine learning models into a dataflow architecture as HLS code, such as HLS4ML and FINN, generally can achieve lower latency by generating customized matrix-vector multipliers and memory structures tailored to the specific fundamental tensor operations required by each layer. However, this approach has significant drawbacks: the compilation process is highly time-consuming and the resulting deployments have unpredictable area and latency, making it impractical to constrain the latency while simultaneously minimizing area. Currently, no existing methods address this type of optimization. In this paper, we present N-TORC (Native Tensor Optimizer for Real-Time Constraints), a novel approach that utilizes data-driven performance and resource models to optimize individual layers of a dataflow architecture. When combined with model hyperparameter optimization, N-TORC can quickly generate architectures that satisfy latency constraints while simultaneously optimizing for both accuracy and resource cost (i.e. offering a set of optimal trade-offs between cost and accuracy). To demonstrate its effectiveness, we applied this framework to a cyber-physical application, DROPBEAR (Dynamic Reproduction of Projectiles in Ballistic Environments for Advanced Research). N-TORC's HLS4ML performance and resource models achieve higher accuracy than prior efforts, and its Mixed Integer Program (MIP)-based solver generates equivalent solutions to a stochastic search in 1000X less time.

Figures (8)

• Figure 1: The DROPEAR experimental setup which consists of a cantilever beam with a movable roller and an accelerometer mounted on the bottom of the beam.
• Figure 2: DROPEAR acceleration data, which results from the roller movements but is treated as an input into a model that predicts the roller location given the acceleration signal.
• Figure 3: DROPEAR roller position, which moves to simulate a moving boundary condition for the cantilever beam. Essentially it sets the root of the cantilever beam.
• Figure 4: LUT cost and latency when scaling size of hardware GEMV unit (as block factor = number of scalar multipliers) for three types of HLS4ML layers. Note that each data point represents a set of observations, as many layer configurations map to the same independent variables shown in the plots (e.g. various combinations of hyperparameters that imply the same values of n_in and n_out); error bars indicate standard deviation.
• Figure 5: Pareto optimal model configurations for accuracy and cost. Included are the positions of Satme et al. network 1 and 2 (purple and green dots) satme2022progress and Kabir et al. (cyan square) kabir2023accelerating.