Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Role Consistency in Multi-Agent Collaboration via Quantitative Role Clarity

Guoling Zhou, Wenpei Han, Fengqin Yang, Li Wang, Yingcong Zhou, Zhiguo Fu

Abstract

In large language model (LLM)-driven multi-agent systems, disobey role specification (failure to adhere to the defined responsibilities and constraints of an assigned role, potentially leading to an agent behaving like another) is a major failure mode \cite{DBLP:journals/corr/abs-2503-13657}. To address this issue, in the present paper, we propose a quantitative role clarity to improve role consistency. Firstly, we construct a role assignment matrix $S(φ)=[s_{ij}(φ)]$, where $s_{ij}(φ)$ is the semantic similarity between the $i$-th agent's behavior trajectory and the $j$-th agent's role description. Then we define role clarity matrix $M(φ)$ as $\text{softmax}(S(φ))-I$, where $\text{softmax}(S(φ))$ is a row-wise softmax of $S(φ)$ and $I$ is the identity matrix. The Frobenius norm of $M(φ)$ quantifies the alignment between agents' role descriptions and their behaviors trajectory. Moreover, we employ the role clarity matrix as a regularizer during lightweight fine-tuning to improve role consistency, thereby improving end-to-end task performance. Experiments on the ChatDev multi-agent system show that our method substantially improves role consistency and task performance: with Qwen and Llama, the role overstepping rate decreases from $46.4\%$ to $8.4\%$ and from $43.4\%$ to $0.2\%$, respectively, and the role clarity score increases from $0.5328$ to $0.9097$ and from $0.5007$ to $0.8530$, respectively, the task success rate increases from $0.6769$ to $0.6909$ and from $0.6174$ to $0.6763$, respectively.

Improving Role Consistency in Multi-Agent Collaboration via Quantitative Role Clarity

Abstract

In large language model (LLM)-driven multi-agent systems, disobey role specification (failure to adhere to the defined responsibilities and constraints of an assigned role, potentially leading to an agent behaving like another) is a major failure mode \cite{DBLP:journals/corr/abs-2503-13657}. To address this issue, in the present paper, we propose a quantitative role clarity to improve role consistency. Firstly, we construct a role assignment matrix , where is the semantic similarity between the -th agent's behavior trajectory and the -th agent's role description. Then we define role clarity matrix as , where is a row-wise softmax of and is the identity matrix. The Frobenius norm of quantifies the alignment between agents' role descriptions and their behaviors trajectory. Moreover, we employ the role clarity matrix as a regularizer during lightweight fine-tuning to improve role consistency, thereby improving end-to-end task performance. Experiments on the ChatDev multi-agent system show that our method substantially improves role consistency and task performance: with Qwen and Llama, the role overstepping rate decreases from to and from to , respectively, and the role clarity score increases from to and from to , respectively, the task success rate increases from to and from to , respectively.

Paper Structure

This paper contains 18 sections, 15 equations, 2 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Quantifiable Role Clarity
  3. Agent Representation
  4. Definition of the Role Clarity
  5. (i) Role descriptions and behavior trajectories embedding vectors.
  6. (ii) Role clarity matrix.
  7. (iii) Role clarity.
  8. LoRA Tuning Based on the Role Clarity Regularization
  9. Preliminaries of Autoregressive Language Models
  10. Optimization Objective with Role Clarity Regularization
  11. Experiments
  12. Experimental Setup
  13. Trajectory Collection
  14. Evaluation Metrics
  15. Experimental Results
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of the LoRA‑tuning framework with role clarity regularization. The framework comprises four stages: (I) collecting high‑quality multi‑turn interaction trajectories via rejection sampling, (II) extracting embeddings and computing role assignment matrix based on similarity, (III) role clarity‑regularized fine tuning using LoRA, and (IV) evaluating role consistency and end-to-end task performance in multi‑agent interactions.
  • Figure 2: Role overstepping rates for DeepSeek Chat, Qwen2.5 7B, and Llama3.1 8B on SWE‑Dev under ChatDev. Lower temperature $T$ yields more deterministic output, whereas higher temperature increases diversity.

Theorems & Definitions (2)

  • Definition 2.1: $\epsilon$-role clear
  • Remark 2.2