Automated Composition of Agents: A Knapsack Approach for Agentic Component Selection

TL;DR

The paper reframes agent composition as a constrained online knapsack problem, where a composer iteratively evaluates candidate components to maximize task success under a budget. It introduces a family of composers (Identity, Retrieval, Offline Knapsack, Online Knapsack) and an online thresholding method (ZCL) to adaptively select components based on real-time utility. Empirical results across single- and multi-agent setups show that online knapsack approaches, especially with AvaTaR optimization, achieve higher success rates at lower costs and maintain Pareto efficiency across diverse domains. The work demonstrates scalable, reuse-friendly assembly of agentic systems and highlights practical considerations for pricing, sandbox testing, and policy-guided tool use. It suggests that combining retrieval with empirical testing and constrained optimization can significantly improve the robustness and efficiency of modular AI ecosystems.

Abstract

Designing effective agentic systems requires the seamless composition and integration of agents, tools, and models within dynamic and uncertain environments. Most existing methods rely on static, semantic retrieval approaches for tool or agent discovery. However, effective reuse and composition of existing components remain challenging due to incomplete capability descriptions and the limitations of retrieval methods. Component selection suffers because the decisions are not based on capability, cost, and real-time utility. To address these challenges, we introduce a structured, automated framework for agentic system composition that is inspired by the knapsack problem. Our framework enables a composer agent to systematically identify, select, and assemble an optimal set of agentic components by jointly considering performance, budget constraints, and compatibility. By dynamically testing candidate components and modeling their utility in real-time, our approach streamlines the assembly of agentic systems and facilitates scalable reuse of resources. Empirical evaluation with Claude 3.5 Sonnet across five benchmarking datasets shows that our online-knapsack-based composer consistently lies on the Pareto frontier, achieving higher success rates at significantly lower component costs compared to our baselines. In the single-agent setup, the online knapsack composer shows a success rate improvement of up to 31.6% in comparison to the retrieval baselines. In multi-agent systems, the online knapsack composer increases success rate from 37% to 87% when agents are selected from an agent inventory of 100+ agents. The substantial performance gap confirms the robust adaptability of our method across diverse domains and budget constraints.

Figures (4)

• Figure 1: The problem of agent composition assumes there is existing inventory of agentic components and a task description. Using these inputs, the agent composition solution should select the most relevant components to build an agent (left). The agentic system can then be deployed as it has the correct set of components to accomplish goals as set by the user at inference.
• Figure 2: Overview of our proposed online knapsack composer. Similar to offline baselines, the workflow begins with generating skills and queries from the given task descriptions (Appendix \ref{['app:skill_prompt']}). Then, the composer retrieves components from the inventory given skill descriptions. With the retrieved components, the composer tests them individually. If value-to-price ratio meets the online knapsack threshold, then it is added as part of the agentic system. Otherwise, the search continues.
• Figure 3: Results of Claude 3.5 Sonnet single-agent experiments where we evaluate all the approaches to select the most relevant tools given task description. After equipping the agent with those tools, we then run evaluation on the dataset and plot success rate against the budget spent on tools. We plot the pareto frontier which represents that no other agent can perform better simultaneously in both success rate and cost. Overall, the online knapsack with AvaTaR optimization ($30) shows to be cost-effective and highest performing approach, followed by online knapsack without optimization ($30).
• Figure 4: Results of Claude 3.5 Sonnet multi-agent experiments where we evaluate all approaches to select most relevant tools given task description. After equipping the agent with those tools, we then run evaluation on the dataset and plot success rate against tool costs. Overall, online knapsack shows to be cost-effective and highest performing approach.