HALOC: Hardware-Aware Automatic Low-Rank Compression for Compact Neural Networks

TL;DR

HALOC introduces a hardware-aware, NAS-inspired framework for automatic low-rank compression of CNNs, recasting layer-wise rank selection as an architectural search problem. It builds an over-parameterized Tucker-2 candidate space per layer, and jointly optimizes candidate weights and per-layer selection probabilities while predicting per-layer latency to enforce hardware constraints. Two design principles—restricting rank candidates to latency-sensitive granularity and enforcing equal-rank Tucker-2 factors—reduce search space, while a feature-map–based regularization mitigates interference between competing rank candidates. Across CIFAR-10 and ImageNet, HALOC delivers substantial FLOPs reductions with minimal or positive accuracy changes and demonstrates practical speedups on desktop GPUs, embedded GPUs, and ASIC accelerators, highlighting its impact for real-world deployment of compact DNNs.

Abstract

Low-rank compression is an important model compression strategy for obtaining compact neural network models. In general, because the rank values directly determine the model complexity and model accuracy, proper selection of layer-wise rank is very critical and desired. To date, though many low-rank compression approaches, either selecting the ranks in a manual or automatic way, have been proposed, they suffer from costly manual trials or unsatisfied compression performance. In addition, all of the existing works are not designed in a hardware-aware way, limiting the practical performance of the compressed models on real-world hardware platforms. To address these challenges, in this paper we propose HALOC, a hardware-aware automatic low-rank compression framework. By interpreting automatic rank selection from an architecture search perspective, we develop an end-to-end solution to determine the suitable layer-wise ranks in a differentiable and hardware-aware way. We further propose design principles and mitigation strategy to efficiently explore the rank space and reduce the potential interference problem. Experimental results on different datasets and hardware platforms demonstrate the effectiveness of our proposed approach. On CIFAR-10 dataset, HALOC enables 0.07% and 0.38% accuracy increase over the uncompressed ResNet-20 and VGG-16 models with 72.20% and 86.44% fewer FLOPs, respectively. On ImageNet dataset, HALOC achieves 0.9% higher top-1 accuracy than the original ResNet-18 model with 66.16% fewer FLOPs. HALOC also shows 0.66% higher top-1 accuracy increase than the state-of-the-art automatic low-rank compression solution with fewer computational and memory costs. In addition, HALOC demonstrates the practical speedups on different hardware platforms, verified by the measurement results on desktop GPU, embedded GPU and ASIC accelerator.

Figures (4)

• Figure 1: The automatic rank selection process of HALOC via alternately updating rank selection probability and model weight.
• Figure 2: The heat map of the measured latency on the practical hardware for Tucker-2 format layers of ResNet-18 with different rank settings. Batch processing is used for stable measurement.
• Figure 3: Approximation error of a decomposed convolutional layer (layer2.1.conv2) in ResNet-18 model with different rank discrepancies ($\Delta Rank=r^{(1)}-r^{(2)}$) and target numbers of parameters after decomposition.
• Figure 4: Experimental analysis for compressing ResNet-18 on ImageNet dataset using HALOC.