Leveraging Neural Graph Compilers in Machine Learning Research for Edge-Cloud Systems

TL;DR

This work addresses the problem of misaligned expectations between academic graph-compiler research and real-world deployment across heterogeneous hardware. It introduces NGraphBench and a pragmatic three-phase research design to systematically evaluate compiler effects on edge-cloud systems, capturing metrics such as $RTR$, $ASE$, and $BSR$ to quantify batch-parallelization, scaling, and friction. Through extensive experiments across CNN and transformer families, the study shows that vendor-specific compilers can provide substantial throughput gains for simple block patterns as depth grows, while automated-tuning approaches like TVM struggle with more complex, hybrid designs. The findings offer practical guidance for deploying neural networks on heterogeneous hardware, highlighting when compiler choices matter most and stressing the need to integrate compiler considerations throughout design, development, and deployment to ensure generalizable, deployment-ready improvements.

Abstract

This work presents a comprehensive evaluation of neural network graph compilers across heterogeneous hardware platforms, addressing the critical gap between theoretical optimization techniques and practical deployment scenarios. We demonstrate how vendor-specific optimizations can invalidate relative performance comparisons between architectural archetypes, with performance advantages sometimes completely reversing after compilation. Our systematic analysis reveals that graph compilers exhibit performance patterns highly dependent on both neural architecture and batch sizes. Through fine-grained block-level experimentation, we establish that vendor-specific compilers can leverage repeated patterns in simple architectures, yielding disproportionate throughput gains as model depth increases. We introduce novel metrics to quantify a compiler's ability to mitigate performance friction as batch size increases. Our methodology bridges the gap between academic research and practical deployment by incorporating compiler effects throughout the research process, providing actionable insights for practitioners navigating complex optimization landscapes across heterogeneous hardware environments.

Figures (12)

• Figure 1: On the Orin, the convolutional-based EfficientNet has higher throughput than the transformer-based Swin. Applying TensorRT significantly improves throughput for both models. However, the EfficientNet is now slower than the Swin. On the Xeon with OpenVINO, we observe the exact inverse behavior.
• Figure 2: Network Architecture Layer Organization
• Figure 3: NGraphBench high-level flow. The client coordinates the experiments for participating clients in a cluster. Each device may evaluate multiple compilers.
• Figure 4: Iterative refinement guided by empirical analysis
• Figure 5: Contrasting the absolute throughput of a vendor-specific compiler with the baseline uncompiled graph. Compiling results in significant throughput gains, except when resources are scarce, the performance saturates.