APEX: Learning Adaptive Priorities for Multi-Objective Alignment in Vision-Language Generation

TL;DR

APEX (Adaptive Priority-based Efficient X-objective Alignment), which stabilizes heterogeneous rewards with Dual-Stage Adaptive Normalization and dynamically schedules objectives via P^3 Adaptive Priorities that combine learning potential, conflict penalty, and progress need, achieves improved Pareto trade-offs across four heterogeneous objectives.

Abstract

Multi-objective alignment for text-to-image generation is commonly implemented via static linear scalarization, but fixed weights often fail under heterogeneous rewards, leading to optimization imbalance where models overfit high-variance, high-responsiveness objectives (e.g., OCR) while under-optimizing perceptual goals. We identify two mechanistic causes: variance hijacking, where reward dispersion induces implicit reweighting that dominates the normalized training signal, and gradient conflicts, where competing objectives produce opposing update directions and trigger seesaw-like oscillations. We propose APEX (Adaptive Priority-based Efficient X-objective Alignment), which stabilizes heterogeneous rewards with Dual-Stage Adaptive Normalization and dynamically schedules objectives via P^3 Adaptive Priorities that combine learning potential, conflict penalty, and progress need. On Stable Diffusion 3.5, APEX achieves improved Pareto trade-offs across four heterogeneous objectives, with balanced gains of +1.31 PickScore, +0.35 DeQA, and +0.53 Aesthetics while maintaining competitive OCR accuracy, mitigating the instability of multi-objective alignment.

Figures (8)

• Figure 1: Images generated by Stable Diffusion 3.5 under different competing objective settings. For the prompt A city square with a billboard... filled with 'Your Ad Here', optimizing for text clarity (left) or visual quality (middle) leads to imbalance. APEX achieves effective multi-objective alignment (right).
• Figure 2: Overview of the APEX framework.Top: The training loop generates rollouts from prompts, evaluates them with multiple reward models, and performs GRPO-based policy optimization. DSAN eliminates variance hijacking via dual-stage normalization, producing balanced gradient contributions. Bottom: The $\mathcal{P}^3$ mechanism analyzes per-objective policy gradients $\nabla_\theta J_k(\theta_t)$ (used only for weight computation, not parameter updates) to compute dynamic weights $w_k^{(t)}$ by fusing learning potential, conflict penalty, and progress need, which are subsequently fed back to DSAN for advantage aggregation.
• Figure 3: Training dynamics revealing variance hijacking. Four subplots track four reward objectives across training steps, comparing APEX (blue) and Static-Weight baseline (red). The baseline shows OCR plateauing while Aesthetic stagnates, whereas APEX achieves balanced growth across all dimensions.
• Figure 4: Analysis of $\bm{\mathcal{P}^3}$ dynamics and Hypervolume progression. (a–c) The three $\mathcal{P}^3$ factors—Learning Potential, Progress Need, and Conflict Penalty—which jointly guide adaptive weight scheduling. (d) Cumulative Hypervolume comparison showing APEX achieves 3.2$\times$ the dominated space volume of the static baseline.
• Figure 5: Ablation study on DSAN. Reward trajectories over the first 400 steps, comparing APEX Full, APEX w/o DSAN, and Static-Weight. Removing DSAN degrades convergence due to variance hijacking, yet the $\mathcal{P}^3$ mechanism still outperforms static weighting. This confirms that DSAN is essential for the adaptive priority mechanism to reach peak performance.

Theorems & Definitions (5)

• Lemma 1: Bounded Priority Factors
• proof
• Corollary 1: Bounded Composite Priority
• Theorem 1: APEX Weight Stability
• proof