AI without networks

TL;DR

This work develops a network-free framework for AI incorporating generative modeling that is theoretically sound, transparent, deterministic, and parameter free, and does not require any compute-expensive training, does not involve optimization, has no model selection, and is easily reproduced and ported.

Abstract

Contemporary Artificial Intelligence (AI) stands on two legs: large training data corpora and many-parameter artificial neural networks (ANNs). The data corpora are needed to represent the complexity and heterogeneity of the world. The role of the networks is less transparent due to the obscure dependence of the network parameters and outputs on the training data and inputs. This raises problems, ranging from technical-scientific to legal-ethical. We hypothesize that a transparent approach to machine learning is possible without using networks at all. By generalizing a parameter-free, statistically consistent data interpolation method, which we analyze theoretically in detail, we develop a network-free framework for AI incorporating generative modeling. We demonstrate this framework with examples from three different disciplines - ethology, control theory, and mathematics. Our generative Hilbert framework applied to the trajectories of small groups of swimming fish outperformed state-of-the-art traditional mathematical behavioral models and current ANN-based models. We demonstrate pure data interpolation based control by stabilizing an inverted pendulum and a driven logistic map around unstable fixed points. Finally, we present a mathematical application by predicting zeros of the Riemann Zeta function, achieving comparable performance as a transformer network. We do not suggest that the proposed framework will always outperform networks as over-parameterized networks can interpolate. However, our framework is theoretically sound, transparent, deterministic, and parameter free: remarkably, it does not require any compute-expensive training, does not involve optimization, has no model selection, and is easily reproduced and ported. We also propose an easily computed method of credit assignment based on this framework, to help address ethical-legal challenges raised by generative AI.

Figures (16)

• Figure 1: Illustration of the Hilbert interpolation scheme in one dimension.a, an example is shown of the Hilbert kernel regression estimator in $d=1$, both within and outside the input data domain. A total of 50 samples $x_i$ were chosen uniformly distributed in the interval $[0.25 ~~0.75]$ and $y_i=\sin(2\pi x_i)+n_i$ with the noise $n_i$ chosen i.i.d. Gaussian distributed $\sim N(0,0.1)$. The sample points are circled, and the function $\sin(2\pi x)$ is shown with a dashed line within the data domain. The solid line is the Hilbert kernel regression estimator. Note the interpolation behavior within the data domain and the extrapolation behavior outside the data domain. b, we plot the results of numerical simulations for the distribution $p$ of the scaling variable $w=\frac{W}{W_n}$, with $W_n\approx\frac{1}{n\ln(n\ln(n))}$, and for $n=65536$ (black line). This is compared to $p(w)=\frac{1}{(1+w)^2}$ (red line), which has the predicted universal tail $p(w)\sim w^{-2}$ for large $w$. c, numerical simulation of the expected value of the Lagrange function of the Hilbert kernel regression estimator in one dimension for a uniform distribution, as in Fig. \ref{['fighilbert']}. $n=400$ samples $x_i$ were chosen uniformly distributed in the interval $[0,1]$ for 100 repeats, and the Lagrange function evaluated at $x_0=0.5$ was averaged across these 100 repeats (blue curve). The black curve shows the asymptotic form $(1+Z)^{-1}$ with $Z=2|x-x_0|/W_n$.
• Figure 1: We report the mean and standard error for the PDF of the observables appearing in Fig. 3 (for $N=2$ individuals) and Fig. 4 (for $N=5$ individuals). The speed $V$ is expressed in cm/s, while the distance to the wall, $r_w$, the distance between two nearest neighbors, $d$, and the gyration radius, $R_{\rm Gyr}$ (only for $N=5$), are expressed in cm. Finally, the polarization, $P$, is without unit and between 0 and 1.
• Figure 1: Classification using the Hilbert kernel: A simple example is shown, with two classes of points drawn from a mixture of 2D unit Normal distributions, with mean separated by 2. The points are shown in green and red colors (1000 points of each class). The red vertical line is the Bayes classification boundary. The yellow and blue colored regions are the Hilbert-predicted classification regions for the green and red points. The islands of blue in yellow (and vice versa) are due to the interpolative nature of the classifier, and correspond to the phenomenon of adversarial examples which are guaranteed for interpolating classifiers on noisy data.
• Figure 2: Relevant fish variables and flow chart of the Hilbert interpolation scheme.a, Relevant variables for an individual: azimuthal angle, $\theta$; heading angle, $\phi$; heading angle relative to the normal to the wall, $\theta_{\rm w}=\phi - \theta$; distance to the wall, $r_{\rm w}$. b, Relevant variables for a pair of individuals: distance between the individuals, $d$; relative heading angle between the 2 individuals, $\Delta\phi$; viewing angle at which the red focal individual perceives the other individual $\psi$. c, Snapshots from experimental videos from papaspyros2023biohybrid, for $N=2$ and $N=5$ fish. d, Flowchart describing the implementation of the Hilbert interpolation scheme (for a memory $M=1$) as a generative model for realistic fish trajectories.
• Figure 2: Behavior of 2 Hilbert fish without the tank wall. This figure is the analog of Fig. 3 in the main text (also for a memory $M=2$), but in the case where the presence of the tank wall is not enforced in the Hilbert model. The different panels show the 9 observables used to characterize the individual ( a- c) and collective ( d- f) behavior, and the time correlations in the system ( g- i): a, PDF of the speed, $V$; b, PDF of the distance to the wall, $r_{\rm w}$; c, PDF of the heading angle relative to the normal to the wall, $\theta_{\rm w}$; d, PDF of the distance between the pair of individuals, $d$; e, PDF of the group polarization, $P=\left|\cos({\Delta\phi}/{2})\right|$, where $\Delta\phi$ is the relative heading angle; f, PDF of the viewing angle at which an individual perceives the other individual, $\psi$. See Fig. \ref{['fig:flowchart']}a and b in the main text for a visual representation of the main variables. g, Mean squared displacement, $C_x(t)$, and its asymptotic limit, $C_x(\infty) =2\langle r^2\rangle$ (dotted lines); h, Velocity autocorrelation, $C_v(t)$; i, Polarization autocorrelation, $C_P(t)$. The black PDFs correspond to experiments, while the red PDFs correspond to the predictions of the Hilbert generative model. The plots are on the same scale as in Fig. 3 in the main text, except for $r_{\rm w}$, for which the horizontal axis has been extended to negative values of $r_{\rm w}$ corresponding to instances where an individual is observed outside the limits of the experimental circular tank. Yet, the Hilbert fish spend 87 % of the time strictly within the tank limits, and when they wander outside the tank, their average excursion distance from the wall is only 1.3 cm. These excursions are responsible for the upward and rightward shift of the peak of $C_x(t)$ and for the larger asymptotic limit, $C_x(\infty) =2\langle r^2\rangle\approx 980$ cm$^2$ (compared to $C_x(\infty)\approx 900$ cm$^2$ for fish or for the Hilbert model implementing the rejection procedure enforcing the presence of the tank wall).

Theorems & Definitions (10)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6
• Theorem 7
• Theorem 8
• Theorem 9
• Theorem 10