Mixed-precision Neural Networks on RISC-V Cores: ISA extensions for Multi-Pumped Soft SIMD Operations

TL;DR

The paper tackles energy-efficient mixed-precision neural network inference on resource-constrained RISC-V cores by introducing three custom ISA instructions, an expanded mixed-precision ALU, multi-pumping, and soft SIMD within a Ibex-based design. It provides an automated co-design framework that performs design-space exploration and evaluates configurations via cycle-accurate emulation, RTL synthesis, and power analysis. The results show substantial gains, with average energy efficiency improvements up to $15\times$ and up to $30\times$ throughput under constrained accuracy budgets, across multiple DNNs and datasets, while incurring modest area/power overhead. This work advances practical deployment of sub-byte DNN quantization on CPU architectures and offers a scalable approach for hardware-aware mixed-precision deployment on edge devices.

Abstract

Recent advancements in quantization and mixed-precision approaches offers substantial opportunities to improve the speed and energy efficiency of Neural Networks (NN). Research has shown that individual parameters with varying low precision, can attain accuracies comparable to full-precision counterparts. However, modern embedded microprocessors provide very limited support for mixed-precision NNs regarding both Instruction Set Architecture (ISA) extensions and their hardware design for efficient execution of mixed-precision operations, i.e., introducing several performance bottlenecks due to numerous instructions for data packing and unpacking, arithmetic unit under-utilizations etc. In this work, we bring together, for the first time, ISA extensions tailored to mixed-precision hardware optimizations, targeting energy-efficient DNN inference on leading RISC-V CPU architectures. To this end, we introduce a hardware-software co-design framework that enables cooperative hardware design, mixed-precision quantization, ISA extensions and inference in cycle-accurate emulations. At hardware level, we firstly expand the ALU unit within our proof-of-concept micro-architecture to support configurable fine grained mixed-precision arithmetic operations. Subsequently, we implement multi-pumping to minimize execution latency, with an additional soft SIMD optimization applied for 2-bit operations. At the ISA level, three distinct MAC instructions are encoded extending the RISC-V ISA, and exposed up to the compiler level, each corresponding to a different mixed-precision operational mode. Our extensive experimental evaluation over widely used DNNs and datasets, such as CIFAR10 and ImageNet, demonstrates that our framework can achieve, on average, 15x energy reduction for less than 1% accuracy loss and outperforms the ISA-agnostic state-of-the-art RISC-V cores.

Figures (8)

• Figure 1: Ibex core architecture showcasing our modified/added features.
• Figure 2: Microarchitecture of the modified ALU featuring packed operations, multi-pumped units and soft SIMD optimizations. Additional multiplier is highlighted in gray. (PPA: Partial Product Addition)
• Figure 3: Illustrative example showing MAC operation optimization and mapping onto four multipliers (M) for three different mixed-precision configurations/modes. Only a partial load of weights and inputs is depicted for simplicity.
• Figure 4: Memory accesses reduction, compared to the original Ibex architecture, delivered by our new instructions when mapping mixed-precision operation . Distinct layers of 3 mixed-quantized MobileNetV1 models were examined.
• Figure 5: Workflow of our co-design framework.