Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MM-tau-p$^2$: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

Anupam Purwar, Aditya Choudhary

TL;DR

This work proposes the MM-tau-p$^2 benchmark with metrics for evaluating the robustness of multi-modal agents in dual control setting with and without persona adaption of user, while also taking user inputs in the planning process to resolve a user query.

Abstract

Current evaluation frameworks and benchmarks for LLM powered agents focus on text chat driven agents, these frameworks do not expose the persona of user to the agent, thus operating in a user agnostic environment. Importantly, in customer experience management domain, the agent's behaviour evolves as the agent learns about user personality. With proliferation of real time TTS and multi-modal language models, LLM based agents are gradually going to become multi-modal. Towards this, we propose the MM-tau-p$^2$ benchmark with metrics for evaluating the robustness of multi-modal agents in dual control setting with and without persona adaption of user, while also taking user inputs in the planning process to resolve a user query. In particular, our work shows that even with state of-the-art frontier LLMs like GPT-5, GPT 4.1, there are additional considerations measured using metrics viz. multi-modal robustness, turn overhead while introducing multi-modality into LLM based agents. Overall, MM-tau-p$^2$ builds on our prior work FOCAL and provides a holistic way of evaluating multi-modal agents in an automated way by introducing 12 novel metrics. We also provide estimates of these metrics on the telecom and retail domains by using the LLM-as-judge approach using carefully crafted prompts with well defined rubrics for evaluating each conversation.

MM-tau-p$^2$: Persona-Adaptive Prompting for Robust Multi-Modal Agent Evaluation in Dual-Control Settings

TL;DR

This work proposes the MM-tau-p$^2 benchmark with metrics for evaluating the robustness of multi-modal agents in dual control setting with and without persona adaption of user, while also taking user inputs in the planning process to resolve a user query.

Abstract

Current evaluation frameworks and benchmarks for LLM powered agents focus on text chat driven agents, these frameworks do not expose the persona of user to the agent, thus operating in a user agnostic environment. Importantly, in customer experience management domain, the agent's behaviour evolves as the agent learns about user personality. With proliferation of real time TTS and multi-modal language models, LLM based agents are gradually going to become multi-modal. Towards this, we propose the MM-tau-p benchmark with metrics for evaluating the robustness of multi-modal agents in dual control setting with and without persona adaption of user, while also taking user inputs in the planning process to resolve a user query. In particular, our work shows that even with state of-the-art frontier LLMs like GPT-5, GPT 4.1, there are additional considerations measured using metrics viz. multi-modal robustness, turn overhead while introducing multi-modality into LLM based agents. Overall, MM-tau-p builds on our prior work FOCAL and provides a holistic way of evaluating multi-modal agents in an automated way by introducing 12 novel metrics. We also provide estimates of these metrics on the telecom and retail domains by using the LLM-as-judge approach using carefully crafted prompts with well defined rubrics for evaluating each conversation.
Paper Structure (32 sections, 1 equation, 3 figures, 5 tables)

This paper contains 32 sections, 1 equation, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Benchmark factors
  4. Domains and tasks.
  5. Critical fields.
  6. Modalities.
  7. Personas and persona exposure.
  8. Dual-control interaction protocol
  9. Voice pipeline and logging
  10. Persona modeling
  11. User Context Injection
  12. Automated scoring with LLM-as-judge
  13. Evaluation Metrics
  14. Goal Achievement Metrics
  15. Efficiency Metrics
  16. ...and 17 more sections

Figures (3)

  • Figure 1: End-to-End Pipeline with conditional edges based on usage (automated vs human-involvement). Automated usage involves use of SOTA TTS and ASR modules for interaction with the Human-Simulator. Texts are tapped into at various stages of the pipeline to generate Ground-Truth and Implementation Transcripts (indicated in yellow)
  • Figure 2: pass1̂ scores for Telecom and Retail domains in Text and Voice Modality. p=0 corresponds to No Persona, p=1 corresponds to Persona Injection and p=2 corresponds to Context Injection
  • Figure 3: pass$^1$ scores segregated by persona difficulty in Telecom domain in Text and Voice Modality. p=0 corresponds to No Persona, p=1 corresponds to Persona Injection and p=2 corresponds to Context Injection