Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Automated Multi-modal Evaluation Framework for Mobile Intelligent Assistants Based on Large Language Models and Multi-Agent Collaboration

Meiping Wang, Jian Zhong, Rongduo Han, Liming Kang, Zhengkun Shi, Xiao Liang, Xing Lin, Nan Gao, Haining Zhang

TL;DR

The paper tackles the scalability and subjectivity challenges of evaluating multi-modal mobile intelligent assistants by introducing a three-tier, LLM-guided evaluation framework that decomposes tasks into interaction, semantic, and experiential assessments. It implements a hierarchical agent architecture with structured scoring (SCQS) and cross-agent reasoning, and optimizes the system through supervised fine-tuning of Qwen3-8B with a loss combining content, consistency, and experience objectives. Across eight cross-platform deployments, the approach achieves high human-model agreement, notably narrowing gaps with strong baselines in the interaction stage, while highlighting ongoing challenges in subjective and cross-modal temporal aspects. This framework advances automated, cross-modal evaluation for mobile assistants, enabling scalable diagnostics and quality assurance with practical implications for product development and benchmarking, albeit with remaining hurdles before fully automated deployment.

Abstract

With the rapid development of mobile intelligent assistant technologies, multi-modal AI assistants have become essential interfaces for daily user interactions. However, current evaluation methods face challenges including high manual costs, inconsistent standards, and subjective bias. This paper proposes an automated multi-modal evaluation framework based on large language models and multi-agent collaboration. The framework employs a three-tier agent architecture consisting of interaction evaluation agents, semantic verification agents, and experience decision agents. Through supervised fine-tuning on the Qwen3-8B model, we achieve a significant evaluation matching accuracy with human experts. Experimental results on eight major intelligent agents demonstrate the framework's effectiveness in predicting users' satisfaction and identifying generation defects.

An Automated Multi-modal Evaluation Framework for Mobile Intelligent Assistants Based on Large Language Models and Multi-Agent Collaboration

TL;DR

The paper tackles the scalability and subjectivity challenges of evaluating multi-modal mobile intelligent assistants by introducing a three-tier, LLM-guided evaluation framework that decomposes tasks into interaction, semantic, and experiential assessments. It implements a hierarchical agent architecture with structured scoring (SCQS) and cross-agent reasoning, and optimizes the system through supervised fine-tuning of Qwen3-8B with a loss combining content, consistency, and experience objectives. Across eight cross-platform deployments, the approach achieves high human-model agreement, notably narrowing gaps with strong baselines in the interaction stage, while highlighting ongoing challenges in subjective and cross-modal temporal aspects. This framework advances automated, cross-modal evaluation for mobile assistants, enabling scalable diagnostics and quality assurance with practical implications for product development and benchmarking, albeit with remaining hurdles before fully automated deployment.

Abstract

With the rapid development of mobile intelligent assistant technologies, multi-modal AI assistants have become essential interfaces for daily user interactions. However, current evaluation methods face challenges including high manual costs, inconsistent standards, and subjective bias. This paper proposes an automated multi-modal evaluation framework based on large language models and multi-agent collaboration. The framework employs a three-tier agent architecture consisting of interaction evaluation agents, semantic verification agents, and experience decision agents. Through supervised fine-tuning on the Qwen3-8B model, we achieve a significant evaluation matching accuracy with human experts. Experimental results on eight major intelligent agents demonstrate the framework's effectiveness in predicting users' satisfaction and identifying generation defects.

Paper Structure

This paper contains 17 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Multi-Agent Evaluation Frameworks
  4. User Experience and Task-Oriented Evaluation
  5. Methodology
  6. Framework Architecture
  7. Multi-Agent Evaluation Mechanism
  8. Interaction Evaluation Agent
  9. Semantic Verification Agent
  10. Experience Decision Agent
  11. Model Optimization
  12. Experiments
  13. Dataset Construction and Annotation
  14. Cross-Platform Evaluation Setup
  15. Model Configuration and Training
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Multi-Agent Evaluation Framework Architecture
  • Figure 2: Dataset analysis showing learning assistance demand and text input dominance
  • Figure 3: Average score performance of major models