DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

TL;DR

DaMo tackles the challenge of selecting optimal data mixtures for multitask fine-tuning of multimodal LLMs used as mobile phone agents. It introduces Downstream Task Performance Prediction (DaPP) via a trainable MLP $f_{MLP}$ that maps data mixture $\mathbf{p}$ and training step $t$ to downstream scores $\hat{\mathbf{s}}$, enabling extrapolation across the fixed data-mixing space $\mathcal{P}_{fix}$. By sampling mixtures, training checkpoints, and evaluating downstream tasks, DaMo learns a predictive surface and identifies the optimal data mixture $\mathbf{p}^*$ to maximize the average downstream performance $S_{\theta}$. The authors also present PhoneAgentBench, a multimodal mobile-application benchmark spanning planning, tool usage, memory, and screen-context understanding across 2,350 test cases. Empirical results show DaMo achieves consistent gains across PhoneAgentBench and standard open benchmarks (BFCL-v3, MME, OCRBench), demonstrates strong generalization, and transfers effectively across models with minimal calibration, offering a practical, scalable solution for on-device multimodal agents.

Abstract

Mobile Phone Agents (MPAs) have emerged as a promising research direction due to their broad applicability across diverse scenarios. While Multimodal Large Language Models (MLLMs) serve as the foundation for MPAs, their effectiveness in handling multiple mobile phone tasks simultaneously remains limited. Although multitask supervised fine-tuning (SFT) is widely adopted for multitask learning, existing approaches struggle to determine optimal training data compositions for peak performance. To address this challenge, we propose DaMo (Data Mixture Optimizer) - a novel solution employing a trainable network that predicts optimal data mixtures by forecasting downstream task performance for any given dataset ratio. To support comprehensive evaluation, we introduce PhoneAgentBench, the first specialized benchmark to evaluate MLLMs on multimodal mobile phone tasks, comprising 1235 QA pairs spanning diverse real-world industrial mobile application scenarios. Demonstrating strong predictive capability (R^2=0.81) in small-scale pilot experiments, DaMo efficiently extrapolates optimal data mixing configurations. Our results show DaMo achieves a 3.38% performance improvement on PhoneAgentBench compared to alternative methods. Furthermore, extensive experiments across established benchmarks including BFCL-v3, MME-Reasoning, MME-Perception, and OCRBench reveal DaMo's superior generalization, outperforming other approaches by 2.57% in terms of average score. When used solely for MLLM optimization on the BFCL-v3 task, DaMo improves the metrics by 12.47% than other methods. Notably, DaMo maintains robust scalability, preserving its effectiveness when applied to other model architectures. The code and dataset are available at https://github.com/OPPO-Mente-Lab/DaMo.git

Figures (13)

• Figure 1: Illustration of our pipeline for obtaining the optimal data mixture. Left: Given $m$ training sets with a batch size of $b$, all possible mixture combinations constitute the data mixing space. We sample a small number of data mixture from this space, train them on a small MLLM, and then evaluate downstream task performance. Using the data mixture as inputs and the metrics as outputs, we fit a MLP to establish the DaMo. By extrapolating from the data mixing space, we predict the optimal data mixture to train the MLLM. Right: Demonstrates the extension and alignment of DaMo to other MLLMs and new data mixing spaces.
• Figure 2: MT-Plan example
• Figure 3: Training dynamics on downstream tasks
• Figure 4: Probability distributions of overall average scores across different checkpoints.
• Figure 5: Scalability Analysis. Top: Scatter plot comparing the predicted overall average scores by the original DaMo against the actual scores of target models. Bottom: Apply linear-mapped correction to DaMo.