Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach

TL;DR

This paper introduces Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a post-training method that leverages self-generated, low-regret decision trajectories to improve large language models as decision-making agents in language-grounded online environments. By repeatedly rolling out decision trajectories, selecting the k-lowest-regret ones, and fine-tuning on them, the approach elicits robust exploration-exploitation behavior and reasoning signals without relying on fixed expert algorithms. Empirical results show improved regret performance and generalization across Transformers with numerical I/O, open-weight LLMs for language-grounded DM, and a closed-weight GPT-4o mini on real-world-context DM tasks, accompanied by theoretical insight that a single-layer Transformer can act as a no-regret learner in a simplified setting. The work highlights a principled, flexible post-training framework that enhances DM capabilities, reduces reliance on hand-crafted prompts, and offers broad potential for real-world DM applications and future extensions to richer environments and longer horizons.

Abstract

Large language models (LLMs) are increasingly deployed as "agents" for decision-making (DM) in interactive and dynamic environments. Yet, since they were not originally designed for DM, recent studies show that LLMs can struggle even in basic online DM problems, failing to achieve low regret or an effective exploration-exploitation tradeoff. To address this, we introduce Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a post-training procedure that repeatedly distills low-regret decision trajectories back into the base model. At each iteration, the model rolls out multiple decision trajectories, selects the k-lowest regret ones, and fine-tunes itself on them. Unlike prior methods that (a) distill action sequences from known DM algorithms or (b) rely on manually crafted chain-of-thought templates, our approach leverages the regret metric to elicit the model's own DM ability and reasoning rationales. This reliance on model-generated reasoning avoids rigid output engineering and provides more flexible, natural-language training signals. Empirical results show that Iterative RMFT improves LLMs' DM performance across diverse models - from Transformers with numerical input/output, to open-weight LLMs, and advanced closed-weight models like GPT-4o mini. Its flexibility in output and reasoning formats enables generalization across tasks with varying horizons, action spaces, reward processes, and natural-language contexts. Finally, we provide theoretical insight showing that a single-layer Transformer under this paradigm can act as a no-regret learner in a simplified setting. Overall, Iterative RMFT offers a principled and general post-training framework for enhancing LLMs' decision-making capabilities.

Figures (32)

• Figure 1: Evolution of the Transformer parameters with different policy spaces. Left:$\ell_2$-ball policy space $\Pi = B(\mathbf{0}_d, 1, \|\cdot\|_2)$ (Env I). Right: Simplex policy space $\Pi = \Delta(\mathcal{A})$ (Env II).
• Figure 2: Evaluation of the trained Transformers with numerical input/output for the MAB environment under Horizon Generalization[$T=25 \rightarrow T=100$], trained and evaluated with the \ref{['gaussian']} reward. The trained model exhibits sublinear regret growth as validated by our validation framework and comparison with baseline algorithms. It also shows $\texttt{MinFrac}(t)$ also exhibits a trend that reflects a proper E-E tradeoff: the metric first increases (active exploration in early rounds) and later decreases (progressive exploitation of optimal actions). Meanwhile, $\texttt{SuffFailFreq}(t)$ maintains a consistently lower value than the Greedy and the Base Model near the end of the horizon, indicating convergence toward more optimal action choices. Overall, these patterns demonstrate an improved E-E tradeoff compared to baselines.
• Figure 3: The regret over time, the final regret distribution, and the metrics of SuffFailFreq$(t)$ and MinFrac$(t)$ for the MAB environment, under both Horizon Generalization [$T=25 \rightarrow T=50$] and Reward Generalization [\ref{['gaussianmu']}$\rightarrow$\ref{['gamma']}] on the Gemma-2-9b-it model, which shows a lower regret value, sublinear regret growth, and improved E-E tradeoff after Iterative RMFT.
• Figure 4: The regret over time, the final regret distribution, and exploration and exploitation metrics using SuffFailFreq$(t)$ and MinFrac$(t)$ for the MAB environment under both Horizon Generalization[$T=25 \rightarrow T=100$] and Reward Generalization[\ref{['gaussianmu']}$\rightarrow$\ref{['gamma']}] on Qwen3-8B, which shows a lower regret value, sublinear regret, and improved E-E tradeoff after Iterative RMFT.
• Figure 5: Illustration of the reasoning rationales generated by the base model and the trained model in the MAB environment for both GPT-4o mini and Qwen3-8B. The figure highlights two major improvements observed after regret-based post-training: (1) enhanced semantic--numerical alignment, and (2) improved E-E tradeoff. Red text indicates incorrect or inconsistent reasoning by the base model, blue text denotes correct and reward-aligned reasoning by the trained model, and purple text provides explanatory annotations clarifying why each response is good or bad.

Theorems & Definitions (7)

• Theorem 1
• Remark 1: Optimal action labels in Iterative RMFT
• Theorem 1
• Claim 1
• Claim 2
• Claim 2
• Claim 2