EPO: Explicit Policy Optimization for Strategic Reasoning in LLMs via Reinforcement Learning

TL;DR

EPO introduces an explicit policy optimization framework for strategic reasoning in LLMs by deploying a dedicated strategic-reasoning LLM (LLM_s) that supplies open-ended strategies to a passive LLM agent (LLM_d). Training uses a lightweight multi-turn reinforcement learning pipeline with process rewards and iterative self-play, enabling robust long-horizon goal alignment without requiring supervised fine-tuning. The approach preserves the generalization of the interacting agent while enabling the strategic model to transfer policies across domains. Empirical results across social and physical tasks show state-of-the-art performance and reveal emergent collaborative reasoning mechanisms, highlighting EPO's potential for real-world strategic decision-making in dynamic environments.

Abstract

Large Language Models (LLMs) have shown impressive reasoning capabilities in well-defined problems with clear solutions, such as mathematics and coding. However, they still struggle with complex real-world scenarios like business negotiations, which require strategic reasoning-an ability to navigate dynamic environments and align long-term goals amidst uncertainty. Existing methods for strategic reasoning face challenges in adaptability, scalability, and transferring strategies to new contexts. To address these issues, we propose explicit policy optimization (EPO) for strategic reasoning, featuring an LLM that provides strategies in open-ended action space and can be plugged into arbitrary LLM agents to motivate goal-directed behavior. To improve adaptability and policy transferability, we train the strategic reasoning model via multi-turn reinforcement learning (RL),utilizing process rewards and iterative self-play. Experiments across social and physical domains demonstrate EPO's ability of long-term goal alignment through enhanced strategic reasoning, achieving state-of-the-art performance on social dialogue and web navigation tasks. Our findings reveal various collaborative reasoning mechanisms emergent in EPO and its effectiveness in generating novel strategies, underscoring its potential for strategic reasoning in real-world applications. Code and data are available at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/EPO.

Figures (4)

• Figure 1: EPO incentivizes goal-directed behavior from LLM agents in interactive scenarios. In such scenarios, each participant's goals and strategies remain private to themselves. Notably, our strategic reasoning model can assist all involved parties, enabling EPO to increase the average payoff for all participants.
• Figure 2: Overview of EPO. The solid line shows the RL training process of the strategic reasoning model, while the dotted line indicates how our reasoning model motivates goal-directed behavior from LLM agents.
• Figure 3: Iterative self-play RL scaling in EPO.Left: The goal completion in test scenarios from SOTOPIA where we use GPT-4o as the self-play dialogue agent. Right: The goal completion of training data for each iteration of RL training. GPT-4-Turbo serves the dialogue agent for self-play in scenarios from SOTOPIA-$\pi$.
• Figure 4: Collaborative reasoning mechanisms in EPO. We evaluate four configurations: (1) “ReAct”, where both dialogue parties generate strategies before responses through prompting; (2) “EPO-RL vs. ReAct”, with one party using an RL-trained reasoning model and the other using ReAct; (3) “EPO-RL vs. EPO-Llama3”, comparing RL-trained and vanilla (Llama3-8B-Instruct) reasoning models; and (4) “EPO-RL”, where both parties employ an RL-trained reasoning model. “Avg Goal” measures the average success in achieving social goals in SOTOPIA. (a) and (c) concern GPT-4o for self-chat, while (b) and (d) involve GPT-3.5-Turbo and GPT-4o as the dialogue partners.