Exploring energy consumption of AI frameworks on a 64-core RV64 Server CPU

TL;DR

This work addresses the energy efficiency of AI inference on emerging multicore RISC-V hardware by benchmarking PyTorch, TensorFlow Lite, and ONNX Runtime on a $64$-core SOPHON SG2042 system across three CNN models (ResNet-50, VGG-16, MobileNetV2). It analyzes time-to-solution and energy-to-solution, highlighting that TensorFlow Lite and ONNX Runtime with XNNPACK back-ends typically outperform PyTorch that relies on OpenBLAS, with notable energy advantages for TF Lite in several cases. The methodology combines hardware-aware benchmarking, diverse back-ends, and a representative image subset (1000 images) to quantify performance and energy across cores and models. The results underscore the importance of software back-end choices for energy efficiency on multicore RISC-V and motivate further hardware-software co-design and broader framework coverage on future systems.

Abstract

In today's era of rapid technological advancement, artificial intelligence (AI) applications require large-scale, high-performance, and data-intensive computations, leading to significant energy demands. Addressing this challenge necessitates a combined approach involving both hardware and software innovations. Hardware manufacturers are developing new, efficient, and specialized solutions, with the RISC-V architecture emerging as a prominent player due to its open, extensible, and energy-efficient instruction set architecture (ISA). Simultaneously, software developers are creating new algorithms and frameworks, yet their energy efficiency often remains unclear. In this study, we conduct a comprehensive benchmark analysis of machine learning (ML) applications on the 64-core SOPHON SG2042 RISC-V architecture. We specifically analyze the energy consumption of deep learning inference models across three leading AI frameworks: PyTorch, ONNX Runtime, and TensorFlow. Our findings show that frameworks using the XNNPACK back-end, such as ONNX Runtime and TensorFlow, consume less energy compared to PyTorch, which is compiled with the native OpenBLAS back-end.

Figures (4)

• Figure 1: Power at rest consumed by the device.
• Figure 2: Resnet-50, VGG-16 and MobileNetV2 inference scaling simulations.
• Figure 3: ResNet-50, VGG-16 and MobileNetV2 inference power consumption, simulations on 1000 images.
• Figure 4: ResNet-50, VGG-16 and MobileNetV2 inference energy consumption, simulations on 1000 images.