Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation

Renhao Li, Minghuan Tan, Derek F. Wong, Min Yang

TL;DR

This paper proposes CoEvol, an LLM-based multi-agent cooperation framework for the improvement of responses for instructions, and develops an iterative framework following a _debate-advise-edit-judge_ paradigm.

Abstract

In recent years, instruction fine-tuning (IFT) on large language models (LLMs) has garnered considerable attention to enhance model performance on unseen tasks. Attempts have been made on automatic construction and effective selection for IFT data. However, we posit that previous methods have not fully harnessed the potential of LLMs for enhancing data quality. The responses within IFT data could be further enhanced by leveraging the capabilities of LLMs themselves. In this paper, we propose CoEvol, an LLM-based multi-agent cooperation framework for the improvement of responses to instructions. To effectively refine the responses, we develop an iterative framework following a debate-advise-edit-judge paradigm. A two-stage multi-agent debate strategy is further devised to ensure the diversity and reliability of editing suggestions within the framework. Empirically, models equipped with CoEvol outperform competitive baselines evaluated by MT-Bench and AlpacaEval, demonstrating its effectiveness in enhancing instruction-following capabilities for LLMs.

CoEvol: Constructing Better Responses for Instruction Finetuning through Multi-Agent Cooperation

TL;DR

This paper proposes CoEvol, an LLM-based multi-agent cooperation framework for the improvement of responses for instructions, and develops an iterative framework following a _debate-advise-edit-judge_ paradigm.

Abstract

In recent years, instruction fine-tuning (IFT) on large language models (LLMs) has garnered considerable attention to enhance model performance on unseen tasks. Attempts have been made on automatic construction and effective selection for IFT data. However, we posit that previous methods have not fully harnessed the potential of LLMs for enhancing data quality. The responses within IFT data could be further enhanced by leveraging the capabilities of LLMs themselves. In this paper, we propose CoEvol, an LLM-based multi-agent cooperation framework for the improvement of responses to instructions. To effectively refine the responses, we develop an iterative framework following a debate-advise-edit-judge paradigm. A two-stage multi-agent debate strategy is further devised to ensure the diversity and reliability of editing suggestions within the framework. Empirically, models equipped with CoEvol outperform competitive baselines evaluated by MT-Bench and AlpacaEval, demonstrating its effectiveness in enhancing instruction-following capabilities for LLMs.
Paper Structure (34 sections, 7 equations, 3 figures, 12 tables, 1 algorithm)

This paper contains 34 sections, 7 equations, 3 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Task Assignment
  4. Evolution Pipeline
  5. Debate Strategy
  6. Predetermined-Position Debate.
  7. Free Debate and Cross-Evaluation.
  8. Data Refinement
  9. Experimental Results
  10. Preliminary Experiment
  11. Experimental Setup
  12. Main Results
  13. Further Experiment
  14. Experimental Setup
  15. Main Results
  16. ...and 19 more sections

Figures (3)

  • Figure 1: Overview of the proposed multi-agent cooperation framework CoEvol.
  • Figure 2: Statistical results of the data evolution process. The proportion of data with different numbers of rounds of evolution driven by CoEvol is shown in figure (a). The average token length of responses in original and evolved data is shown in figure (b). We report statistical results on different datasets and backbone LLMs for agents.
  • Figure 3: Overview of the evolving direction of CoEvol. Due to space constraints, we merge similar root verbs and show parts of verb-object pairs with top counts.