MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization

TL;DR

MAESTRO tackles open-domain LLM alignment by reframing reward scalarization as a context-dependent decision. A lightweight Conductor network uses terminal hidden states to select among $K=5$ reward components, forming a dynamic scalarization that feeds a GRPO-based inner loop while a meta-objective updates the Conductor with group-relative advantages in an asynchronous bi-time-scale scheme. This contextual bandit and bi-level setup enables co-evolution of adaptively weighted evaluative criteria and generation strategies, yielding stronger multi-objective performance across seven open-domain benchmarks with preserved or improved training efficiency. The approach demonstrates that reward composition should be context-aware rather than fixed, enabling flexible balancing of rigor, creativity, and format validity in open-ended generation.

Abstract

Group-Relative Policy Optimization (GRPO) has emerged as an efficient paradigm for aligning Large Language Models (LLMs), yet its efficacy is primarily confined to domains with verifiable ground truths. Extending GRPO to open-domain settings remains a critical challenge, as unconstrained generation entails multi-faceted and often conflicting objectives - such as creativity versus factuality - where rigid, static reward scalarization is inherently suboptimal. To address this, we propose MAESTRO (Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization), which introduces a meta-cognitive orchestration layer that treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck to perceive task-specific priorities. We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal. Across seven benchmarks, MAESTRO consistently outperforms single-reward and static multi-objective baselines, while preserving the efficiency advantages of GRPO, and in some settings even reducing redundant generation.

Figures (8)

• Figure 1: Comparison of reward orchestration strategies. (a) Rule-based methods, limited to verifiable ground truths. (b) LLM-as-a-Judge, flexible but with high computational overhead. (c) Static multi-objective optimization using fixed, prompt-independent reward weights across all contexts. (d) MAESTRO (Ours), which dynamically adapts reward weights according to prompt--response semantics.
• Figure 2: Architecture of the MAESTRO framework. Given a prompt $q$, the policy model $\pi_\theta$ samples a group of candidate outputs $\{o_i\}$. The Conductor $\pi_\phi$ simultaneously processes prompt--response hidden states to sample a reward-emphasis action $a$, inducing a weighting scheme $\mathbf{w}^{(a)}$. Raw reward vectors $\mathbf{r}$ and KL penalties are fused by the scalarization node ($\Sigma$) to compute the scalar reward $R$, which is normalized via group computation to obtain group-relative advantages $\hat{A}$. MAESTRO adopts bi-level optimization: $\pi_\theta$ is updated with GRPO, while $\pi_\phi$ performs meta-updates using group advantages as reinforcement signals.
• Figure 3: Comparison of learned reward weight allocation dynamics in the course of training across two datasets. Both plots use the same linear training-step axis to facilitate direct comparison.
• Figure 4: Illustrative prompt template modification applied to Qwen3-8B.
• Figure 5: Illustrative evaluation prompt for creative writing tasks.