Kernel Learning for Regression via Quantum Annealing Based Spectral Sampling

TL;DR

The paper addresses regression with kernel methods by learning a data-adaptive shift-invariant kernel through its spectral distribution, using quantum annealing as a sampler for an RBM-modeled spectrum. It builds random Fourier features via a Gaussian–Bernoulli mapping from QA-generated RBM states and trains the kernel end-to-end by minimizing a leave-one-out Nadaraya--Watson loss with squared-kernel weights for stability. Empirically, the learned kernel reshapes the kernel matrix and generally improves $R^2$ and RMSE over a fixed Gaussian kernel, with further gains from increasing the number of random features and, at inference, from endpoint local linear regression. The work demonstrates that QA-based sampling can be integrated into a complete kernel-learning pipeline to yield data-adaptive kernels for regression, with future potential in scaling spectral models and extending to classification and uncertainty-aware tasks.

Abstract

While quantum annealing (QA) has been developed for combinatorial optimization, practical QA devices operate at finite temperature and under noise, and their outputs can be regarded as stochastic samples close to a Gibbs--Boltzmann distribution. In this study, we propose a QA-in-the-loop kernel learning framework that integrates QA not merely as a substitute for Markov-chain Monte Carlo sampling but as a component that directly determines the learned kernel for regression. Based on Bochner's theorem, a shift-invariant kernel is represented as an expectation over a spectral distribution, and random Fourier features (RFF) approximate the kernel by sampling frequencies. We model the spectral distribution with a (multi-layer) restricted Boltzmann machine (RBM), generate discrete RBM samples using QA, and map them to continuous frequencies via a Gaussian--Bernoulli transformation. Using the resulting RFF, we construct a data-adaptive kernel and perform Nadaraya--Watson (NW) regression. Because the RFF approximation based on $\cos(\bmω^{\top}Δ\bm{x})$ can yield small negative values and cancellation across neighbors, the Nadaraya--Watson denominator $\sum_j k_{ij}$ may become close to zero. We therefore employ nonnegative squared-kernel weights $w_{ij}=k(\bm{x}_i,\bm{x}_j)^2$, which also enhances the contrast of kernel weights. The kernel parameters are trained by minimizing the leave-one-out NW mean squared error, and we additionally evaluate local linear regression with the same squared-kernel weights at inference. Experiments on multiple benchmark regression datasets demonstrate a decrease in training loss, accompanied by structural changes in the kernel matrix, and show that the learned kernel tends to improve $R^2$ and RMSE over the baseline Gaussian-kernel NW. Increasing the number of random features at inference further enhances accuracy.

Figures (5)

• Figure 1: Training loss trajectory and comparison with the NW loss. The loss decreases as training proceeds.
• Figure 2: Kernel matrix on the bodyfat dataset. Training changes off-diagonal kernel components $k_{ij}$, indicating kernel adaptation.
• Figure 3: Comparison of $R^2$ versus the number of random features $S$ at inference. Red: before training, blue: after training, green: NW baseline. Solid lines: training, dashed lines: test.
• Figure 4: Comparison of RMSE versus the number of random features $S$ at inference. Red: before training, blue: after training, green: NW baseline. Solid lines: training, dashed lines: test.
• Figure 5: Histogram of sampled frequencies $\omega$ on the ccs dataset ($S=1000$). The learned distribution differs from a single Gaussian.