Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MARFT: Multi-Agent Reinforcement Fine-Tuning

Junwei Liao, Muning Wen, Jun Wang, Weinan Zhang

TL;DR

<3-5 sentence high-level summary> MARFT introduces a novel reinforcement-fine-tuning framework for LaMAS that integrates RFT, MARL, and flexible multi-agent governance via Flex-MG. It formalizes action- and token-level MARFT variants (MARFT-A and MARFT-T) and provides a base algorithm with LoRA-based adapters, validated on math and coding tasks showing improved multi-agent collaboration over single-agent baselines. The work discusses perspectives on scalability, privacy, and blockchain integration, and outlines open problems in dynamic environments, data efficiency, and unified frameworks. Overall, MARFT offers a principled path toward scalable, adaptive, and cooperative agentic AI systems that preserve pretrained capabilities while enabling task-specific coordination.

Abstract

LLM-based Multi-Agent Systems have demonstrated remarkable capabilities in addressing complex, agentic tasks, from generating high-quality presentation slides to even conducting sophisticated scientific research. Meanwhile, RL has been widely recognized for its effectiveness in enhancing agent intelligence, but limited research has investigated the fine-tuning of LaMAS using foundational RL techniques. Moreover, the direct application of MARL methods to LaMAS introduces significant challenges, stemming from the unique characteristics and mechanisms inherent to LaMAS. To address these challenges, this article presents a comprehensive study of LLM-based MARL and proposes a novel paradigm termed Multi-Agent Reinforcement Fine-Tuning (MARFT). We introduce a brand-new MG called Flex-MG, which aligns with the LaMAS optimization in real-world applications and a universal algorithmic framework tailored specifically for LaMAS, outlining the conceptual foundations, key distinctions, and practical implementation strategies. We review the evolution from RL to RFT, setting the stage for a parallel analysis in the multi-agent domain. In the context of LaMAS, we elucidate critical differences between MARL and MARFT. These differences motivate a transition toward a LaMAS-oriented formulation of RFT. Central to this work is a robust and scalable MARFT framework. We detail the core algorithm and provide a complete, open-source implementation to facilitate adoption and further research. The latter sections of the paper explore real-world application perspectives and opening challenges in MARFT. By bridging theoretical underpinnings with practical methodologies, this work serves as a roadmap for researchers seeking to advance MARFT toward resilient and adaptive solutions in agentic systems. Our implementation of the proposed framework is publicly available at: https://github.com/jwliao-ai/MARFT.

MARFT: Multi-Agent Reinforcement Fine-Tuning

TL;DR

<3-5 sentence high-level summary> MARFT introduces a novel reinforcement-fine-tuning framework for LaMAS that integrates RFT, MARL, and flexible multi-agent governance via Flex-MG. It formalizes action- and token-level MARFT variants (MARFT-A and MARFT-T) and provides a base algorithm with LoRA-based adapters, validated on math and coding tasks showing improved multi-agent collaboration over single-agent baselines. The work discusses perspectives on scalability, privacy, and blockchain integration, and outlines open problems in dynamic environments, data efficiency, and unified frameworks. Overall, MARFT offers a principled path toward scalable, adaptive, and cooperative agentic AI systems that preserve pretrained capabilities while enabling task-specific coordination.

Abstract

LLM-based Multi-Agent Systems have demonstrated remarkable capabilities in addressing complex, agentic tasks, from generating high-quality presentation slides to even conducting sophisticated scientific research. Meanwhile, RL has been widely recognized for its effectiveness in enhancing agent intelligence, but limited research has investigated the fine-tuning of LaMAS using foundational RL techniques. Moreover, the direct application of MARL methods to LaMAS introduces significant challenges, stemming from the unique characteristics and mechanisms inherent to LaMAS. To address these challenges, this article presents a comprehensive study of LLM-based MARL and proposes a novel paradigm termed Multi-Agent Reinforcement Fine-Tuning (MARFT). We introduce a brand-new MG called Flex-MG, which aligns with the LaMAS optimization in real-world applications and a universal algorithmic framework tailored specifically for LaMAS, outlining the conceptual foundations, key distinctions, and practical implementation strategies. We review the evolution from RL to RFT, setting the stage for a parallel analysis in the multi-agent domain. In the context of LaMAS, we elucidate critical differences between MARL and MARFT. These differences motivate a transition toward a LaMAS-oriented formulation of RFT. Central to this work is a robust and scalable MARFT framework. We detail the core algorithm and provide a complete, open-source implementation to facilitate adoption and further research. The latter sections of the paper explore real-world application perspectives and opening challenges in MARFT. By bridging theoretical underpinnings with practical methodologies, this work serves as a roadmap for researchers seeking to advance MARFT toward resilient and adaptive solutions in agentic systems. Our implementation of the proposed framework is publicly available at: https://github.com/jwliao-ai/MARFT.

Paper Structure

This paper contains 72 sections, 22 equations, 10 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Reinforcement Fine-Tuning (RFT)
  4. Language-augmented Markov Decision Process
  5. Reinforcement Learning (RL)
  6. The Objective of RL
  7. Value-based Methods
  8. Policy-based Methods
  9. Actor-Critic
  10. From RL to RFT
  11. Differences
  12. Implementations
  13. Prevailing Objective Functions
  14. LLM-based Multi-Agent Systems (LaMAS)
  15. Multi-Agent Reinforcement Learning (MARL)
  16. ...and 57 more sections

Figures (10)

  • Figure 1: Illustration of MARFT in real-world agentic problem-solving scenarios.
  • Figure 2: A detailed illustration of the dynamics of a Flex-MG. The dependency function (dashed purple line) can vary across timesteps.
  • Figure 3: The procedure of Multi-Agent Reinforcement Fine-Tuning. Inference and training are conducted in an alternating manner. Each agent within the MAS can have its own private APIs, tool pool, database, and other resources.
  • Figure 4: Learning dynamics of MARFT-A of Duo on MATH and CMATH.
  • Figure 5: Learning dynamics of MARFT-A on CodeForces.
  • ...and 5 more figures