Adapting to Covariate Shift in Real-time by Encoding Trees with Motion Equations

TL;DR

The study tackles covariate shift in online settings by proposing Xenovert, a motion-equation–driven, perfect binary tree that adaptively partitions the input space into quasi-quantile intervals $q_i$ to maintain alignment between source and shifted target distributions without retraining. Xenovert updates the quasi-quantiles via $q_{i,t+1} = q_{i,t} + \alpha v_{i,t+1} s$ with $v_{i,t+1} = \theta v_{i,t} + |q_{i,t}-x_t|$, propagating updates from the root to leaves and converting inputs to interval indices, yielding an $O(NL)$ complexity. When integrated with a neural network, Xenovert improved robustness to covariate shift, delivering best performance in 4 of 5 shifted datasets and preserving accuracy on severely shifted Iris data, while also reducing regression error in Abalone tasks compared to a plain MLP. The approach offers a simple, online alternative to reweighting methods, with potential extensions to high-dimensional inputs and broader applications in systems facing unforeseen distribution changes. Overall, Xenovert provides a practical, low-overhead mechanism for maintaining model relevance under non-stationary environments by continuously adapting input representations without full retraining.

Abstract

Input distribution shift presents a significant problem in many real-world systems. Here we present Xenovert, an adaptive algorithm that can dynamically adapt to changes in input distribution. It is a perfect binary tree that adaptively divides a continuous input space into several intervals of uniform density while receiving a continuous stream of input. This process indirectly maps the source distribution to the shifted target distribution, preserving the data's relationship with the downstream decoder/operation, even after the shift occurs. In this paper, we demonstrated how a neural network integrated with Xenovert achieved better results in 4 out of 5 shifted datasets, saving the hurdle of retraining a machine learning model. We anticipate that Xenovert can be applied to many more applications that require adaptation to unforeseen input distribution shifts, even when the distribution shift is drastic.

Figures (7)

• Figure 1: Xenovert's adaptation from source distribution (red) to target distribution (blue). The histogram displays the frequency of interval activation, indicating that all intervals have nearly identical input frequencies. Thus, this shows that the quasi-quantiles can divide the input distribution uniformly.
• Figure 2: Illustration of indexing and level structure in Xenovert. Xenovert divides the input using a tree structure with each node setting boundaries into quasi-quantile, with leaf nodes indexing these quasi-quantiles. The index of quasi-quantiles, $i = \{0, 1, \cdots, n\}$ are sorted from left to right in ascending order regardless of the hierarchy. The levels are sorted in descending order in a top-down manner.
• Figure 3: The illustration of Xenovert's update function. Xenovert is a perfect binary tree that uses its nodes to represent quasi-quantiles $q$, which divide the input space into equal-sized intervals. In detail, when given an input $x$ at time step $t$, the chosen quasi-quantiles approximate towards the input with the update function $q_{i,t+1} = q_{i,t} + \alpha v_{i,t+1}s$; Where $v$ is a velocity term, $alpha$ is the learning rate, $\theta$ is the velocity decay, and $s=-1$ if $q-x>0$, otherwise, $s=1$. Starting from the root quasi-quantiles $q_r$, we select the 'left' child node if $x < q_r$ (or $x < q$, when it is not a root node); otherwise, select the 'right' child node. The selected child node is marked in black. The gray shade is the interval that the input falls into. If the input distribution remains constant, the quasi-quantiles converge to an equilibrium state, eventually ensuring that all intervals contain an equal concentration of inputs.
• Figure 4: Simulation result of distributions with three types of shifting. (A) The ridgeline plot shows the shifting from source distribution to target distribution (bottom to top). The bottom row shows the degree of shifting in source and target distribution. Based on the HI score learning curve, we observed a general trend where the HI score drops when shifting starts to occur and returns to a near-optimum score when Xenovert adapts to the new distribution. (B) The adaptation of Xenovert is invariant to the degree of shifting. Firstly, we experimented with Xenovert's adaptation using inputs drawn from a normal distribution, $\mathcal{N}(0, 5)$ and $\mathcal{N}(50, 5)$ as the source and target distributions, respectively. Followed by that, we experimented with another set of distributions, $\mathcal{N}(0, 5)$ and $\mathcal{N}(250, 5)$. The HI score learning curve, however, remains almost identical in both experiments, proving its invariance to the degree of shifting. (C) The higher the total level, the lesser the data compression and information lost, but the lower the HI score. (D) The stability-adaptability trade-off, where the higher the learning rate, the lower the stability but the faster the adaptability.
• Figure 5: Diagram illustrating the integration of Xenoverts with a Neural Network. The dataset consists of $p$ features, and each feature is independently processed by one of the $p$ Xenoverts, resulting in a set of quantized inputs. These inputs are then fed into the neural network. As the Xenoverts adapt, the neural network is concurrently trained. Notably, after a covariate shift, the neural network's parameters can remain frozen, performing inference as usual. However, the Xenoverts continue to adapt to the evolving input distribution, ensuring the quantized inputs stay pertinent for the neural network's inference.