Facilitating Trustworthy Human-Agent Collaboration in LLM-based Multi-Agent System oriented Software Engineering
Krishna Ronanki
TL;DR
This work addresses trustworthy human–agent collaboration in LLM-based multi-agent software engineering by proposing a RACI-based Responsibility Assignment Framework. It grounds the approach in Trustworthy AI principles and adapts RACI to assign Responsible, Accountable, Consulted, and Informed roles across humans and LLM agents. The framework provides step-by-step guidelines and an example DevOps implementation to produce verifiable artefacts while ensuring regulatory alignment. Although theoretical, it lays the foundation for empirical validation, aiming to improve safety, accountability, and efficiency in LMA-oriented SE workflows.
Abstract
Multi-agent autonomous systems (MAS) are better at addressing challenges that spans across multiple domains than singular autonomous agents. This holds true within the field of software engineering (SE) as well. The state-of-the-art research on MAS within SE focuses on integrating LLMs at the core of autonomous agents to create LLM-based multi-agent autonomous (LMA) systems. However, the introduction of LMA systems into SE brings a plethora of challenges. One of the major challenges is the strategic allocation of tasks between humans and the LMA system in a trustworthy manner. To address this challenge, a RACI-based framework is proposed in this work in progress article, along with implementation guidelines and an example implementation of the framework. The proposed framework can facilitate efficient collaboration, ensure accountability, and mitigate potential risks associated with LLM-driven automation while aligning with the Trustworthy AI guidelines. The future steps for this work delineating the planned empirical validation method are also presented.