Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MELTing point: Mobile Evaluation of Language Transformers

Stefanos Laskaridis, Kleomenis Katevas, Lorenzo Minto, Hamed Haddadi

TL;DR

This paper introduces MELT, a mobile-edge benchmarking framework for automated, fine-grained evaluation of LLMs on diverse devices. It conducts a systematic study across Android, iOS, and Nvidia Jetson platforms to measure throughput, energy, memory usage, and QoE, using a model zoo with quantization and multiple backends. Key findings show that on-device LLM inference is largely memory-bound, quantization dramatically reduces memory footprints but incurs accuracy costs, and continuous edge execution is challenged by energy and thermal constraints; offloading and NPU-enabled co-design are promising directions. The work provides a practical foundation and dataset for optimizing on-device LLMs, highlighting the need for hardware-software co-design to enable private, responsive, and sustainable edge intelligence.

Abstract

Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with "sparks of intelligence". However, their runtime requirements have prevented them from being broadly deployed on mobile. As personal devices become increasingly powerful and prompt privacy becomes an ever more pressing issue, we explore the current state of mobile execution of Large Language Models (LLMs). To achieve this, we have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device, supporting different models, devices and frameworks, including Android, iOS and Nvidia Jetson devices. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance, tracing their memory and energy requirements along the way. Our analysis is the first systematic study of on-device LLM execution, quantifying performance, energy efficiency and accuracy across various state-of-the-art models and showcases the state of on-device intelligence in the era of hyperscale models. Results highlight the performance heterogeneity across targets and corroborates that LLM inference is largely memory-bound. Quantization drastically reduces memory requirements and renders execution viable, but at a non-negligible accuracy cost. Drawing from its energy footprint and thermal behavior, the continuous execution of LLMs remains elusive, as both factors negatively affect user experience. Last, our experience shows that the ecosystem is still in its infancy, and algorithmic as well as hardware breakthroughs can significantly shift the execution cost. We expect NPU acceleration, and framework-hardware co-design to be the biggest bet towards efficient standalone execution, with the alternative of offloading tailored towards edge deployments.

MELTing point: Mobile Evaluation of Language Transformers

TL;DR

This paper introduces MELT, a mobile-edge benchmarking framework for automated, fine-grained evaluation of LLMs on diverse devices. It conducts a systematic study across Android, iOS, and Nvidia Jetson platforms to measure throughput, energy, memory usage, and QoE, using a model zoo with quantization and multiple backends. Key findings show that on-device LLM inference is largely memory-bound, quantization dramatically reduces memory footprints but incurs accuracy costs, and continuous edge execution is challenged by energy and thermal constraints; offloading and NPU-enabled co-design are promising directions. The work provides a practical foundation and dataset for optimizing on-device LLMs, highlighting the need for hardware-software co-design to enable private, responsive, and sustainable edge intelligence.

Abstract

Transformers have revolutionized the machine learning landscape, gradually making their way into everyday tasks and equipping our computers with "sparks of intelligence". However, their runtime requirements have prevented them from being broadly deployed on mobile. As personal devices become increasingly powerful and prompt privacy becomes an ever more pressing issue, we explore the current state of mobile execution of Large Language Models (LLMs). To achieve this, we have created our own automation infrastructure, MELT, which supports the headless execution and benchmarking of LLMs on device, supporting different models, devices and frameworks, including Android, iOS and Nvidia Jetson devices. We evaluate popular instruction fine-tuned LLMs and leverage different frameworks to measure their end-to-end and granular performance, tracing their memory and energy requirements along the way. Our analysis is the first systematic study of on-device LLM execution, quantifying performance, energy efficiency and accuracy across various state-of-the-art models and showcases the state of on-device intelligence in the era of hyperscale models. Results highlight the performance heterogeneity across targets and corroborates that LLM inference is largely memory-bound. Quantization drastically reduces memory requirements and renders execution viable, but at a non-negligible accuracy cost. Drawing from its energy footprint and thermal behavior, the continuous execution of LLMs remains elusive, as both factors negatively affect user experience. Last, our experience shows that the ecosystem is still in its infancy, and algorithmic as well as hardware breakthroughs can significantly shift the execution cost. We expect NPU acceleration, and framework-hardware co-design to be the biggest bet towards efficient standalone execution, with the alternative of offloading tailored towards edge deployments.
Paper Structure (28 sections, 12 figures, 6 tables)

This paper contains 28 sections, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background & Motivation
  3. Transformer Preliminaries
  4. Large Language Models
  5. Current State and Motivating Factors
  6. MELT Infrastructure
  7. PhoneLab
  8. JetsonLab
  9. Methodology
  10. Model Zoo and Evaluation
  11. Automated On-Device Benchmarking
  12. Evaluation
  13. Experimental Setup
  14. Macro Experiments
  15. Dataset Qualitative Analysis
  16. ...and 13 more sections

Figures (12)

  • Figure 1: Architecture and workflow of MELT
  • Figure 2: MELT (Experiment Process)
  • Figure 3: Qualitative analysis of prompts used for macro-experiments to assess the behaviour of LLM-powered chats on device.
  • Figure 4: Throughput across frameworks and devices
  • Figure 5: Discharge per token across frameworks and devices. Missing bars indicate unsuccessful runs due to OOM errors or too long runtime ($>$1hr per conversation).
  • ...and 7 more figures