Neural Contextual Bandits with Deep Representation and Shallow Exploration
Pan Xu, Zheng Wen, Handong Zhao, Quanquan Gu
TL;DR
The paper introduces Neural-LinUCB, a neural contextual bandit method that decouples representation learning from exploration by transforming contexts with the deep network’s last hidden layer and performing UCB exploration only in the final linear layer. It provides a sublinear regret guarantee of order $\tilde{O}(\sqrt{T})$ under NTK-based assumptions and shows that the approach remains computationally efficient by avoiding exploration across all network parameters. Theoretical guarantees hinge on a combination of linear-bandit-style analysis and neural-network NTK properties, with an error term that depends on how well the neural network approximates the true reward function. Empirically, Neural-LinUCB outperforms NeuralUCB and Neural-Linear on real datasets while achieving substantially faster runtimes, highlighting the practical value of deep representation with shallow exploration.
Abstract
We study a general class of contextual bandits, where each context-action pair is associated with a raw feature vector, but the reward generating function is unknown. We propose a novel learning algorithm that transforms the raw feature vector using the last hidden layer of a deep ReLU neural network (deep representation learning), and uses an upper confidence bound (UCB) approach to explore in the last linear layer (shallow exploration). We prove that under standard assumptions, our proposed algorithm achieves $\tilde{O}(\sqrt{T})$ finite-time regret, where $T$ is the learning time horizon. Compared with existing neural contextual bandit algorithms, our approach is computationally much more efficient since it only needs to explore in the last layer of the deep neural network.