Serving LLMs in HPC Clusters: A Comparative Study of Qualcomm Cloud AI 100 Ultra and NVIDIA Data Center GPUs

TL;DR

This work benchmarks the Qualcomm Cloud AI 100 Ultra (QAic) against NVIDIA A100 GPUs for large language model inference on the National Research Platform (NRP), focusing on energy efficiency, throughput, and resource granularity under multi-tenant workloads. Using 12 open-source LLMs and the vLLM serving framework, the study measures performance and power across 1–8 QAic devices and 4x/8x A100 configurations, with detailed hardware and software pipelines including QPC compilation and ONNX exports. Key findings show QAic can achieve substantial power savings and finer resource granularity (e.g., serving 70B models with 1–2 QAic devices) while A100s deliver higher absolute throughput; the results quantify trade-offs and guide deployment decisions in energy-constrained HPC environments. The paper demonstrates that purpose-built inference accelerators like QAic can complement GPU-based deployments by enabling cost-effective, scalable inference in multi-tenant research platforms where power, cooling, and resource fragmentation are critical considerations.

Abstract

This study presents a benchmarking analysis of the Qualcomm Cloud AI 100 Ultra (QAic) accelerator for large language model (LLM) inference, evaluating its energy efficiency (throughput per watt), performance, and hardware scalability against NVIDIA A100 GPUs (in 4x and 8x configurations) within the National Research Platform (NRP) ecosystem. A total of 12 open-source LLMs, ranging from 124 million to 70 billion parameters, are served using the vLLM framework. Our analysis reveals that QAic achieves competitive energy efficiency with advantages on specific models while enabling more granular hardware allocation: some 70B models operate on as few as 1 QAic card versus 8 A100 GPUs required, with 20x lower power consumption (148W vs 2,983W). For smaller models, single QAic devices achieve up to 35x lower power consumption compared to our 4-GPU A100 configuration (36W vs 1,246W). The findings offer insights into the potential of the Qualcomm Cloud AI 100 Ultra for energy-constrained and resource-efficient HPC deployments within the National Research Platform (NRP).