An Empirical Study of On-Device Translation for Real-Time Live-Stream Chat on Mobile Devices

TL;DR

This work tackles the pragmatic deployment of on-device translation for real-time mobile live-stream chat by examining model selection, resource usage, and domain adaptation. It introduces LiveChatBench, a 1,000-sentence Korean–English benchmark reflecting memes and slang, and evaluates three small-to-mid parametric models across five iOS and two Android devices under CPU-only and GPU-accelerated conditions. The results indicate that domain-adapted on-device translators can reach performance close to GPT-5.1 on a targeted task, despite hardware constraints and limited data coverage, highlighting substantial potential for latency-sensitive on-device AI. Nevertheless, gaps remain on longer, broader-domain benchmarks, signaling room for improvements in generalization, data curation, and end-to-end mobile deployment realism.

Abstract

Despite its efficiency, there has been little research on the practical aspects required for real-world deployment of on-device AI models, such as the device's CPU utilization and thermal conditions. In this paper, through extensive experiments, we investigate two key issues that must be addressed to deploy on-device models in real-world services: (i) the selection of on-device models and the resource consumption of each model, and (ii) the capability and potential of on-device models for domain adaptation. To this end, we focus on a task of translating live-stream chat messages and manually construct LiveChatBench, a benchmark consisting of 1,000 Korean-English parallel sentence pairs. Experiments on five mobile devices demonstrate that, although serving a large and heterogeneous user base requires careful consideration of highly constrained deployment settings and model selection, the proposed approach nevertheless achieves performance comparable to commercial models such as GPT-5.1 on the well-targeted task. We expect that our findings will provide meaningful insights to the on-device AI community.

Figures (7)

• Figure 1: Mobile device temperature results on LiveChatBench across different on-device models.
• Figure 2: CPU utilization results on LiveChatBench across different on-device models.
• Figure 3: Heatmap of six types of translation errors. Darker colors indicate a higher number of errors.
• Figure 4: Comparison of the translation outputs of GPT-5.1 and trained Gemma-3-270M.
• Figure 5: Heatmap of six types of translation errors (FLORES-200). Darker colors indicate a higher number of errors.