Agent Skill Acquisition for Large Language Models via CycleQD

TL;DR

CycleQD addresses the challenge of acquiring multiple agent skills in LLMs by combining Quality Diversity with cyclic task optimization. It uses a model merging crossover and an SVD-based mutation to fuse expert-task models into a cohesive multi-skill agent without data ratio tuning or misaligned objectives. Across CS tasks on the Llama-3-8B-Instruct backbone, CycleQD matches or surpasses fine-tuning baselines and approaches GPT-3.5-TURBO on coding, OS, and DB tasks, while preserving language abilities; it also generalizes to VQA and image segmentation. This work demonstrates the potential of evolutionary QD approaches to scalable, cross-domain skill acquisition for LLMs.

Abstract

Training large language models to acquire specific skills remains a challenging endeavor. Conventional training approaches often struggle with data distribution imbalances and inadequacies in objective functions that do not align well with task-specific performance. To address these challenges, we introduce CycleQD, a novel approach that leverages the Quality Diversity framework through a cyclic adaptation of the algorithm, along with a model merging based crossover and an SVD-based mutation. In CycleQD, each task's performance metric is alternated as the quality measure while the others serve as the behavioral characteristics. This cyclic focus on individual tasks allows for concentrated effort on one task at a time, eliminating the need for data ratio tuning and simplifying the design of the objective function. Empirical results from AgentBench indicate that applying CycleQD to LLAMA3-8B-INSTRUCT based models not only enables them to surpass traditional fine-tuning methods in coding, operating systems, and database tasks, but also achieves performance on par with GPT-3.5-TURBO, which potentially contains much more parameters, across these domains. Crucially, this enhanced performance is achieved while retaining robust language capabilities, as evidenced by its performance on widely adopted language benchmark tasks. We highlight the key design choices in CycleQD, detailing how these contribute to its effectiveness. Furthermore, our method is general and can be applied to image segmentation models, highlighting its applicability across different domains.

Figures (8)

• Figure 1: Method Overview. CycleQD uses QD in a cyclic manner to merge LLMs. Archives are initialized with expert LLMs, each of which has been fine-tuned to excel in a specific task. Model merging is conducted using QD, treating task performance as quality (Q) and behavior characteristics (BC), which are cyclically swapped in each generation.
• Figure 2: MAP-Elites flow.
• Figure 3: SVD-based mutations. Left: A task vector $\tau$ contains multiple parameter matrices $\tau_l$. E.g., the query projection matrix from the attention block in layer 1, the key projection matrix from the attention block in layer 2, etc. Those that have a rank $r>1$ can be decomposed using SVD into $r$ components. Right: SVD-based mutation samples a vector $w \in \mathbb{R}^r$ and scales each component by $w_i$, essentially adding perturbations that are aligned with the "directions" of the components.
• Figure 4: CycleQD Archives from Computer Science Tasks. In each archive, the two axes show the BCs, and the color intensities indicate the LLM agent's quality in that grid. These archives are obtained after 1200 generations of CycleQD. The red bounding boxes indicate the grids where expert models were present. See Appendix \ref{['sec:development_of_archives']} for archive development across generations.
• Figure 5: Averaged performance on computer science tasks. Evaluated every 100 generations.