DEPO: Dual-Efficiency Preference Optimization for LLM Agents

TL;DR

DEPO defines dual-efficiency for LLM agents as joint optimization of step-level (minimizing tokens per interaction) and trajectory-level (minimizing total steps) efficiency. It introduces DEPO, an offline preference-learning method that extends vanilla KTO with an efficiency bonus added to the log-ratio and a KL regularization term, enabling training without reward models or on-policy sampling. Through MCTS-generated data, BC fine-tuning, and offline preference optimization, DEPO achieves substantial token and step reductions (up to 60.9% and 26.9%, respectively) while improving performance (up to 29.3%) on Webshop and BabyAI, and shows strong generalization to GSM8K, MATH, and SimulEq, even with limited data (25%). These results highlight DEPO’s practical potential for deploying fast, accurate LLM agents in real-world, interactive environments.

Abstract

Recent advances in large language models (LLMs) have greatly improved their reasoning and decision-making abilities when deployed as agents. Richer reasoning, however, often comes at the cost of longer chain of thought (CoT), hampering interaction efficiency in real-world scenarios. Nevertheless, there still lacks systematic definition of LLM agent efficiency, hindering targeted improvements. To this end, we introduce dual-efficiency, comprising (i) step-level efficiency, which minimizes tokens per step, and (ii) trajectory-level efficiency, which minimizes the number of steps to complete a task. Building on this definition, we propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps. Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance. DEPO also generalizes to three out-of-domain math benchmarks and retains its efficiency gains when trained on only 25% of the data. Our project page is at https://opencausalab.github.io/DEPO.

Figures (4)

• Figure 1: A comparison between (a) step-level inefficiency, arising from latency and cost in LLM token generation; (b) trajectory-level inefficiency, arising from latency and cost in environment interactions such as API calls, and our defined (c) dual-efficiency. For LLM agents, achieving genuine efficiency requires joint optimization across both dimensions.
• Figure 2: Model generalizability across math benchmarks. The left y‑axis shows accuracy and the right y‑axis shows average tokens. Qwen refers to Qwen2.5-7B-Instruct, Qwen+ to Qwen2.5-7B-BC+DEPO, Llama to Llama3.1-8B-Instruct, Llama+ to Llama3.1-8B-BC+DEPO.
• Figure 3: Sample efficiency of DEPO. Relative improvement $\triangle$% over the BC baseline for four metrics as the training‐set proportion increases from 25% to 100%.
• Figure 4: Desirable bonus only (DEPO) vs. Desirable bonus (DEPO) + Undesirable penalty. Each point shows the relative change of DEPO+ undesirable penalty against the DEPO, with $\triangle$% = (Penalty $-$DEPO)/DEPO × 100.