Probabilistic Hash Embeddings for Online Learning of Categorical Features

TL;DR

The paper tackles learning with open-vocabulary categorical features in streaming data, where the item vocabulary changes over time. It introduces Probabilistic Hash Embeddings (PHE), a Bayesian online learning framework that treats hash embeddings as random variables and updates their posteriors incrementally, using a shared embedding table and multiple hash functions to bound memory. The authors provide scalable variational inference methods (variational EM) and show that Bayes online learning matches batch performance regardless of item arrival order, while avoiding catastrophic forgetting. Across classification, sequence modeling, and large-scale recommendation, PHE outperforms deterministic hash-based baselines and closely tracks or surpasses collision-free upper bounds with far lower memory usage. The results demonstrate PHE’s practical impact for real-time systems that must adapt to new categories without growing memory or sacrificing accuracy, making it a valuable plug-in for probabilistic models in dynamic tabular data domains.

Abstract

We study streaming data with categorical features where the vocabulary of categorical feature values is changing and can even grow unboundedly over time. Feature hashing is commonly used as a pre-processing step to map these categorical values into a feature space of fixed size before learning their embeddings. While these methods have been developed and evaluated for offline or batch settings, in this paper we consider online settings. We show that deterministic embeddings are sensitive to the arrival order of categories and suffer from forgetting in online learning, leading to performance deterioration. To mitigate this issue, we propose a probabilistic hash embedding (PHE) model that treats hash embeddings as stochastic and applies Bayesian online learning to learn incrementally from data. Based on the structure of PHE, we derive a scalable inference algorithm to learn model parameters and infer/update the posteriors of hash embeddings and other latent variables. Our algorithm (i) can handle an evolving vocabulary of categorical items, (ii) is adaptive to new items without forgetting old items, (iii) is implementable with a bounded set of parameters that does not grow with the number of distinct observed values on the stream, and (iv) is invariant to the item arrival order. Experiments in classification, sequence modeling, and recommendation systems in online learning setups demonstrate the superior performance of PHE while maintaining high memory efficiency (consumes as low as 2~4 memory of a one-hot embedding table). Supplementary materials are at https://github.com/aodongli/probabilistic-hash-embeddings

Figures (11)

• Figure 1: On two tabular datasets, Mushroom and Adult, we split the data into groups based on random partitions of categorical column vocabularies, such that each group has disjoint vocabulary sets. We report the results before and after online learning on each group. The performance gaps motivate the need to learn representations of new items. Bracketed values indicate the splitting columns. Results are averaged over five runs. Partition details are in \ref{['fig:group-info']}.
• Figure 2: PHE architecture. The input ${\mathbf{s}}$ can be usernames or anonymized strings. The entire module represents $p(E_{{\mathbf{h}}_{\mathbf{s}}})$.
• Figure 3: (left) Online classification results on Adult tabular data streams. In the parentheses is the column whose items embeddings get updated. The Ada results show a downward trend although there are no new items to learn, suggesting the deterministic hash embeddings suffer from forgetting during the learning. In contrast, the proposed PHE mitigates the forgetting issue and keeps performing as good as the upper-bound method P-EE. Other datasets in \ref{['fig:all-tab-online']} in \ref{['sec:app-classification']} show similar conclusions. (right) Results of sequence modeling on Retail data-streams. It shows that PHE outperforms all Ada baselines that are sensitive to their optimization hyperparameters. Moreover, it is remarkable to note that PHE performs slightly better than the collision-free P-EE baseline, especially considering PHE consumes only $2\%$ of the memory of P-EE.
• Figure 4: A graphical model of a temporal sequence with PHE. The changing categorical values are contained in ${\mathbf{h}}_{\mathbf{s}}$.
• Figure 5: Forgetting in online learning using deterministic hash embedding on synthetic data. (The complete setting is described in \ref{['sec:minor_theory']}.) The task is predicting a scalar (regression problem) with the covariate being a categorical variable that takes one of two values of ${\mathbf{m}}_0$ or ${\mathbf{m}}_1$. a) shows the embedding matrix $E$ of size $3 \times 1$. Here the number of buckets $B=3$ and $d=1$. The two hash function maps ${\mathbf{m}}_0$ to $0$ and $1$ respectively and maps ${\mathbf{m}}_1$ to $1$ and $2$ respectively. b) shows the online samples where the covariate alternates between ${\mathbf{m}}_0$ and ${\mathbf{m}}_1$ and the corresponding target $y({\mathbf{m}}_i)$ takes values in $1$ and $-1$. c) shows the prediction of a probabilistic hash embedding table (blue) trained using Bayesian online learning and a deterministic hash embedding (DHE) table (yellow) trained using online gradient descent. d) plots the prediction error. From these figures we observe that PHE's prediction error converges to $0$ much quicker than DHE. After every $20$ samples when the covariate changes, there is a big jump in DHE error, exhibiting forgetting while the PHE has no error spikes after it has encountered both the categorical values.

Theorems & Definitions (2)

• Proposition 3.1
• proof