Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Domain Adaptive Neural Contextual Bandits

Ziyan Wang, Xiaoming Huo, Hao Wang

TL;DR

This paper addresses cross-domain contextual bandits where target-domain feedback is costly by introducing DABand, which jointly explores in a low-cost source domain and aligns feature representations with a high-cost target domain. The authors formulate rigorous cross-domain theory, extending regression-based domain adaptation to the bandit setting with a sub-linear target-regret bound that decomposes into source regret, regression error, data divergence, and a predicted-reward regularization term. They operationalize the bound via a differentiable, minimax training objective that combines representation learning, adversarial alignment, and LinUCB-style exploration, yielding the DABand algorithm. Empirical results on DIGIT, VisDA17, and S2RDA49 show substantial improvements over strong baselines, highlighting the method’s ability to transfer knowledge across domains while controlling exploration costs. The work lays a foundation for online, domain-adaptive decision-making with potential applications in drug testing, robotics, and other cost-sensitive domains.

Abstract

Contextual bandit algorithms are essential for solving real-world decision making problems. In practice, collecting a contextual bandit's feedback from different domains may involve different costs. For example, measuring drug reaction from mice (as a source domain) and humans (as a target domain). Unfortunately, adapting a contextual bandit algorithm from a source domain to a target domain with distribution shift still remains a major challenge and largely unexplored. In this paper, we introduce the first general domain adaptation method for contextual bandits. Our approach learns a bandit model for the target domain by collecting feedback from the source domain. Our theoretical analysis shows that our algorithm maintains a sub-linear regret bound even adapting across domains. Empirical results show that our approach outperforms the state-of-the-art contextual bandit algorithms on real-world datasets.

Towards Domain Adaptive Neural Contextual Bandits

TL;DR

This paper addresses cross-domain contextual bandits where target-domain feedback is costly by introducing DABand, which jointly explores in a low-cost source domain and aligns feature representations with a high-cost target domain. The authors formulate rigorous cross-domain theory, extending regression-based domain adaptation to the bandit setting with a sub-linear target-regret bound that decomposes into source regret, regression error, data divergence, and a predicted-reward regularization term. They operationalize the bound via a differentiable, minimax training objective that combines representation learning, adversarial alignment, and LinUCB-style exploration, yielding the DABand algorithm. Empirical results on DIGIT, VisDA17, and S2RDA49 show substantial improvements over strong baselines, highlighting the method’s ability to transfer knowledge across domains while controlling exploration costs. The work lays a foundation for online, domain-adaptive decision-making with potential applications in drug testing, robotics, and other cost-sensitive domains.

Abstract

Contextual bandit algorithms are essential for solving real-world decision making problems. In practice, collecting a contextual bandit's feedback from different domains may involve different costs. For example, measuring drug reaction from mice (as a source domain) and humans (as a target domain). Unfortunately, adapting a contextual bandit algorithm from a source domain to a target domain with distribution shift still remains a major challenge and largely unexplored. In this paper, we introduce the first general domain adaptation method for contextual bandits. Our approach learns a bandit model for the target domain by collecting feedback from the source domain. Our theoretical analysis shows that our algorithm maintains a sub-linear regret bound even adapting across domains. Empirical results show that our approach outperforms the state-of-the-art contextual bandit algorithms on real-world datasets.
Paper Structure (39 sections, 8 theorems, 34 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 39 sections, 8 theorems, 34 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theory
  4. Preliminaries
  5. Our Cross-Domain Contextual Bandit Setting
  6. Formal Definitions of Error
  7. Source Regret and Target Regret
  8. Cross-Domain Error Bound for Regression
  9. Final Regret Bound
  10. Method
  11. From Theory to Practice: Translating the Bound in Eqn. (\ref{['eq:total_bound']}) to Differentiable Loss Terms
  12. Putting Everything Together: DABand Training Algorithm
  13. Experiments
  14. Conclusion
  15. Proofs
  16. ...and 24 more sections

Key Result

Theorem 3.1

Denoting the contexts from the source domain ${\mathcal{S}}$ as $\{ {\bf x}_{t, a}^{\mathcal{S}} \}_{i \in [N], a \in [K]}$, the associated ground-truth action as $\{ a_{i}^{*} \}_{i=1}^{N}$, and the contexts from the target domain ${\mathcal{T}}$ as $\{ {\bf x}_{t, a}^{\mathcal{T}} \}_{i \in [N], a where $\mathit{R}_{\mathcal{S}}$, $\epsilon_{\mathcal{S}} (h)$, and $\widehat{d}_{\mathcal{H}\Delta

Figures (2)

  • Figure 1: The cumulative regrets of different methods for 1,920 rounds on DIGIT (left), VisDA17 (middle), and S2RDA49 (right). Results are averaged over 5 runs. LinUCB is not reported for VisDA17 and S2RDA49 due to out-of-memory issues.
  • Figure 2: Sub-linearity for each term in Eqn. (\ref{['eq:total_bound']}).

Theorems & Definitions (22)

  • Definition 3.1: Labeling Function and Hypothesis
  • Definition 3.2: Estimated Error between Two Hypotheses
  • Definition 3.3: Source- and Target-Domain Error
  • Definition 3.4: Source Regret
  • Definition 3.5: Target Regret and Problem Formulation
  • Definition 3.6: $\mathcal{H}\Delta\mathcal{H}$ Hypothesis Space for Regression
  • Definition 3.7: Optimal Hypothesis
  • Theorem 3.1: Target Regret Bound
  • Lemma A.1
  • proof
  • ...and 12 more