DARA: Few-shot Budget Allocation in Online Advertising via In-Context Decision Making with RL-Finetuned LLMs

TL;DR

This work tackles few-shot budget allocation in online advertising under AI-Generated Bidding by proposing DARA, a dual-phase architecture that separates high-level planning (Few Shot Reasoner) from fine-grained optimization (Fine-grained Optimizer). It introduces GRPO-Adaptive, an RL fine-tuning strategy that periodically refreshes the KL-regularization reference to stabilize learning while enhancing reasoning and numerical precision. The approach is evaluated in real-world and synthetic environments, showing consistent reductions in marginal ROI variance and outperforming strong baselines, with the greatest gains arising from the dual-phase + RL combination. Overall, DARA enables robust, interpretable, and data-efficient decision making for budget allocation in dynamic online advertising settings, with practical implications for improved ROI under budget constraints.

Abstract

Optimizing the advertiser's cumulative value of winning impressions under budget constraints poses a complex challenge in online advertising, under the paradigm of AI-Generated Bidding (AIGB). Advertisers often have personalized objectives but limited historical interaction data, resulting in few-shot scenarios where traditional reinforcement learning (RL) methods struggle to perform effectively. Large Language Models (LLMs) offer a promising alternative for AIGB by leveraging their in-context learning capabilities to generalize from limited data. However, they lack the numerical precision required for fine-grained optimization. To address this limitation, we introduce GRPO-Adaptive, an efficient LLM post-training strategy that enhances both reasoning and numerical precision by dynamically updating the reference policy during training. Built upon this foundation, we further propose DARA, a novel dual-phase framework that decomposes the decision-making process into two stages: a few-shot reasoner that generates initial plans via in-context prompting, and a fine-grained optimizer that refines these plans using feedback-driven reasoning. This separation allows DARA to combine LLMs' in-context learning strengths with precise adaptability required by AIGB tasks. Extensive experiments on both real-world and synthetic data environments demonstrate that our approach consistently outperforms existing baselines in terms of cumulative advertiser value under budget constraints.

Figures (6)

• Figure 1: Data Environment.
• Figure 2: The training workflow of DARA.
• Figure 3: Performance of different algorithms in reducing marginal ROI variance.
• Figure 4: Impact of dual-phase architecture and RL fine-tuning on marginal ROI variance.
• Figure 5: Experimental results with different numbers of periods.