Compound AI Systems Optimization: A Survey of Methods, Challenges, and Future Directions

TL;DR

This paper tackles optimization of compound AI systems—composed of LLMs, tools, and retrieval modules—where many components are non-differentiable. It introduces a graph-based formalism, $\\Phi=(G,\\mathcal{F})$, with $G=(V,E)$ and conditional edges $c_{ij}$ that govern runtime topology, enabling end-to-end analysis of interactions. A 2×2 taxonomy over Structural Flexibility and Learning Signals is proposed to classify 26 surveyed works into four quadrants, guiding principled comparisons and future directions. The authors synthesize key trends, open challenges (e.g., computation, safety, library standardization), and provide an open repository to track developments in this rapidly evolving field.

Abstract

Recent advancements in large language models (LLMs) and AI systems have led to a paradigm shift in the design and optimization of complex AI workflows. By integrating multiple components, compound AI systems have become increasingly adept at performing sophisticated tasks. However, as these systems grow in complexity, new challenges arise in optimizing not only individual components but also their interactions. While traditional optimization methods such as supervised fine-tuning (SFT) and reinforcement learning (RL) remain foundational, the rise of natural language feedback introduces promising new approaches, especially for optimizing non-differentiable systems. This paper provides a systematic review of recent progress in optimizing compound AI systems, encompassing both numerical and language-based techniques. We formalize the notion of compound AI system optimization, classify existing methods along several key dimensions, and highlight open research challenges and future directions in this rapidly evolving field. A list of surveyed papers is publicly available at https://github.com/MiuLab/AISysOpt-Survey.

Figures (5)

• Figure 1: The proposed 2$\times$2 taxonomy spans Structural Flexibility (y-axis) and Learning Signals (x-axis). Representative methods for each quadrant along with their key designs and potential drawbacks.
• Figure 2: Example of a compound AI system and its optimization. Centered on LLMs and coupled with multiple interacting modules, the system handles complex user queries. Automated optimization strategies leverage two types of learning signals, i.e., natural language feedback and numerical signals (defined in Sec. \ref{['sec:four_dim']}), to backpropagate errors and guide system updates toward improved performance.
• Figure 3: Learning Signals are classified into two categories, with Numerical Signals further divided by their utilization schemes: (a) one class of methods devises rule-based algorithms that directly learn from raw system performance metrics, and (b) another class transforms system evaluation results into formalized training objectives. These objectives are further split as (b1) supervised fine-tuning (SFT) losses, (b2) reinforcement learning (RL) reward functions, and (b3) direct preference optimization (DPO) DPO losses.
• Figure 4: The frequency statistics of benchmarks tested in the surveyed 26 papers.
• Figure 5: (a) During optimization, the algorithm explores the design space defined by user constraints and algorithmic parameters (Sec. \ref{['sec:four_dim']}). Although the conditional arguments in the edge matrix $E = [c_{ij}]$ are fixed once optimization completes, the actual on/off status of each $c_{ij}$ remains undetermined. (b) At runtime, the optimized $\Phi$ instantiates different execution topologies based on $c_{ij}(\tau)$, reflecting dependence on the query input $q_i$ and the induced contextual state $\tau$.