Learning with the $p$-adics

TL;DR

The paper investigates learning in the $p$-adic world $\mathbb{Q}_p$ as an alternative to real-valued frameworks, highlighting the inherent hierarchies of $p$-adic balls and their suitability for classification, regression, and representation learning. It develops concrete learning rules: unidimensional and multidimensional $p$-adic classifiers correspond to enclosing balls and unions of balls, while regression becomes exemplar-based due to ultrametricity; it also demonstrates a small-scale semantic-network example using Quillian-like networks with compact $p$-adic embeddings. The work provides algorithmic approaches (e.g., $p$-adic beam search) and theoretical results about what $p$-adic models can and cannot compute (e.g., XOR solvability in low dimensions, congruence and counting). Open problems include native multi-class extensions, gradient-based $p$-adic optimization via Hensel lifting, deeper architectures, and adelic predictors across all primes, suggesting a rich future research program in non-Archimedean machine learning.

Abstract

Existing machine learning frameworks operate over the field of real numbers ($\mathbb{R}$) and learn representations in real (Euclidean or Hilbert) vector spaces (e.g., $\mathbb{R}^d$). Their underlying geometric properties align well with intuitive concepts such as linear separability, minimum enclosing balls, and subspace projection; and basic calculus provides a toolbox for learning through gradient-based optimization. But is this the only possible choice? In this paper, we study the suitability of a radically different field as an alternative to $\mathbb{R}$ -- the ultrametric and non-archimedean space of $p$-adic numbers, $\mathbb{Q}_p$. The hierarchical structure of the $p$-adics and their interpretation as infinite strings make them an appealing tool for code theory and hierarchical representation learning. Our exploratory theoretical work establishes the building blocks for classification, regression, and representation learning with the $p$-adics, providing learning models and algorithms. We illustrate how simple Quillian semantic networks can be represented as a compact $p$-adic linear network, a construction which is not possible with the field of reals. We finish by discussing open problems and opportunities for future research enabled by this new framework.

Figures (6)

• Figure 1: Hierarchical structure of $\mathbb{Q}_p$ for $p=2$.
• Figure 2: Hierarchical structure of $\mathbb{Q}_p$ for $p=2$. Balls in $\mathbb{Q}_p$ are either nested or disjoint and each ball is associated to a node in the hierarchy. Shown are two nested balls, $\bar{B}_1(0)$, which is the set of $p$-adic integers, and $\bar{B}_{1/4}(4)$, which is contained in the former and whose elements are $p$-adic integers of the form $\textcolor{red}{\cdots 10}$.
• Figure 3: Left: A separable dataset in $\mathbb{Q}_2$ and enclosing balls. The positive examples are of the form *110 and the negative examples either of the form *00 or *1. We choose $x_i=6 = \textcolor{red}{110}$ as representative of the positive class. The point $x_j=-2 = \textcolor{red}{\cdots 1110}$ is maximally distant from $x_i$. Using Proposition \ref{['prop:binary_classifier_conditions']}, we obtain $w^\star = 1/8$ and $b^\star = -3/4$, corresponding to the minimal enclosing ball $\bar{B}_{1/8}(6) = \textcolor{red}{*110}$. The maximal enclosing ball is $\bar{B}_{1/4}(6) = \textcolor{red}{*10}$. Right: Adding the outlier $x_- = \textcolor{red}{\cdots 11001110}$ turns the dataset not separable. Let the dataset be $\mathcal{D}_+ = \{\textcolor{red}{*0110}, \textcolor{red}{*01001110}, \textcolor{red}{*11110}\}$ and $\mathcal{D}_- = \{\textcolor{red}{*00000}, \textcolor{red}{*10000}, \textcolor{red}{*100}, \textcolor{red}{*11001110}, \textcolor{red}{*1}\}$ (the left continuation is not important). We represent these points in a tree where each leaf represents the largest ball containing a single point and the nodes represent splitting points (edges are labeled with one or more symbols). To determine the classifier with the smallest misclassification error, we consider a ball rooted at each node (or leaf) and compute the training error associated with that ball. In this example, the optimal classifier corresponds to either of the balls $\bar{B}_{1/4}(2) = \textcolor{red}{*10}$ or $\bar{B}_{1/8}(6) = \textcolor{red}{*110}$, where the dashed edge is cut, leading to a training error of 1/8.
• Figure 4: Example of a 2nd order classifier with $f(x) = \frac{1}{64}(x-1)(x-5)$. The positive region is the union $\bar{B}_{\frac{1}{16}}(1) \cup \bar{B}_{\frac{1}{16}}(5)$.
• Figure 5: Top: Semantic network adapted from quillian1968semantic. Bottom left: Neural network developed by rumelhart1990brain and mcclelland1995there to answer queries using this semantic model. Given the active inputs "robin can", the network produces the completion "grow move fly". Note the two (non-linear) hidden layers, the first of which embeds input entities onto $\mathbb{R}^6$. Bottom right: A linear $p$-adic network with a single embedding dimension ($\mathbb{Q}_p$) which solves the same problem (colors and leaves attributes are excluded for simplicity).

Theorems & Definitions (18)

• proposition 1: $p$-adic expansion
• proposition 2
• proposition 3: 2nd order $p$-adic classifier
• proposition 4: Properties of $\mathbb{Q}_p$-linear classifiers
• proposition 5
• remark 1: Reduction to integers
• remark 2: Integer weights
• proposition 6
• proposition 7
• proposition 8