Provably Efficient Multi-Objective Bandit Algorithms under Preference-Centric Customization

TL;DR

This work introduces Preference-Aware MO-MAB (PAMO-MAB), where each user carries a D-dimensional preference vector and learning aims to maximize cumulative inner-product rewards within the Pareto front rather than achieving blanket Pareto optimality. The authors design a two-component framework—preference estimation and preference-aware optimization—addressing unknown and hidden user preferences. They propose two algorithms: PRUCB-HP for the hidden-preference setting, featuring a weighted least-squares preference estimator and a dual-exploration policy with reward and preference bonuses; and PRUCB-UP for the provided-preference case, with a simplified estimator and optimization that achieves near-optimal regret. Theoretical results establish sublinear regret bounds under both scenarios, and extensive numerical analyses show strong empirical gains over traditional MO-MAB baselines, validating effective online preference learning and customized optimization within the Pareto front. Overall, the work offers provable performance guarantees for personalized MO-MAB and demonstrates practical effectiveness for preference-centric customization in multi-objective decision problems.

Abstract

Multi-objective multi-armed bandit (MO-MAB) problems traditionally aim to achieve Pareto optimality. However, real-world scenarios often involve users with varying preferences across objectives, resulting in a Pareto-optimal arm that may score high for one user but perform quite poorly for another. This highlights the need for customized learning, a factor often overlooked in prior research. To address this, we study a preference-aware MO-MAB framework in the presence of explicit user preference. It shifts the focus from achieving Pareto optimality to further optimizing within the Pareto front under preference-centric customization. To our knowledge, this is the first theoretical study of customized MO-MAB optimization with explicit user preferences. Motivated by practical applications, we explore two scenarios: unknown preference and hidden preference, each presenting unique challenges for algorithm design and analysis. At the core of our algorithms are preference estimation and preference-aware optimization mechanisms to adapt to user preferences effectively. We further develop novel analytical techniques to establish near-optimal regret of the proposed algorithms. Strong empirical performance confirm the effectiveness of our approach.

Figures (9)

• Figure 1: A scenario of users interacting with a conversational recommender for restaurant recommendation. (a) Recommender achieves Pareto optimality but receives low rating from user. (b) Recommendations with high users' ratings when the recommender captures users' preferences and aligns optimization with preferences.
• Figure 2: A scenario of user’s preference feedback is not explicitly provided (hidden preference).
• Figure 3: A 2-dimensional hidden preference PAMO-MAB toy example with mean preference $\overline{\boldsymbol{c}} = [0.5, 0.5]$, illustrating preference estimate $\hat{\boldsymbol{c}}$ via linear regression using reward data from (a) Arm-1 (dominated mean reward: $[0.2,0.2]$) and (b) Arm-2 (Pareto-optimal mean reward: $[0.8,0.8]$).
• Figure 4: An example where user explicitly provides her preference to LLM system before (❶) or after (❷) the response of movie recommendation (decision making).
• Figure 5: Regret comparison of our proposed PRUCB with other benchmarks under different preference environments, where our methods outperforms other methods significantly.

Theorems & Definitions (49)

• Definition 1: Preference-Free Algorithm
• Proposition 1
• Lemma 1
• Lemma 2
• Lemma 3
• Theorem 1
• Remark 1
• Theorem 2
• Remark 2
• Lemma 4: Variant of Lemma 7 in jun2018adversarial