FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations

TL;DR

This work introduces FlashMem, a memory streaming framework designed to efficiently execute large-scale modern DNNs and multi-DNN workloads while minimizing memory consumption and reducing inference latency, and statically determines model loading schedules and dynamically streams them on demand.

Abstract

The increasing size and complexity of modern deep neural networks (DNNs) pose significant challenges for on-device inference on mobile GPUs, with limited memory and computational resources. Existing DNN acceleration frameworks primarily deploy a weight preloading strategy, where all model parameters are loaded into memory before execution on mobile GPUs. We posit that this approach is not adequate for modern DNN workloads that comprise very large model(s) and possibly execution of several distinct models in succession. In this work, we introduce FlashMem, a memory streaming framework designed to efficiently execute large-scale modern DNNs and multi-DNN workloads while minimizing memory consumption and reducing inference latency. Instead of fully preloading weights, FlashMem statically determines model loading schedules and dynamically streams them on demand, leveraging 2.5D texture memory to minimize data transformations and improve execution efficiency. Experimental results on 11 models demonstrate that FlashMem achieves 2.0x to 8.4x memory reduction and 1.7x to 75.0x speedup compared to existing frameworks, enabling efficient execution of large-scale models and multi-DNN support on resource-constrained mobile GPUs.

Figures (10)

• Figure 1: Left (a) shows the hierarchical memory structure and bandwidth on mobile GPUs, highlighting the multi-step weight transfer path from disk to streaming multiprocessors. Right (b and c) depicts two multi-DNN support strategies.
• Figure 2: Latency increase (ms) across different operators on mobile GPUs due to additional data transformations. The X-axis shows the ratio of additional data loaded compared to the original kernel input (e.g., 1.0 indicates an equal amount of extra data is streamed alongside computation). Threshold markers show where latency overhead reaches 20% and 30% of the original kernel.
• Figure 3: Overview of FlashMem.
• Figure 4: Profiling operators from more than ten models. We systematically adjust key parameters in our framework to measure kernel latency changes. The collected data is then used to train our XGBoost-based regression model for accurate latency prediction. Global Work Size (GWS) and Local Work Size (LWS) determine workload distribution and thread partition, respectively.
• Figure 5: Comparison of a naive Matrix Multiplication kernel with the rewritten kernel integrated with weight loading.