Learning Regularities from Data using Spiking Functions: A Theory

TL;DR

The paper introduces a theory that defines non-randomness in data through spiking functions and formalizes regularities as concise, information-rich encodings. By leveraging information-theoretic metrics like KL-divergence and spiking efficiency SE_f, it connects the discovery of structure to the compression of information into small parameter counts via A_f = SE_f · C_f. The framework extends to multiple spiking functions and Gamma-level optimal encoders, offering a principled path to learn and compare encoders that capture large amounts of information with minimal representation. A practical, layer-wise bi-output learning pipeline is proposed to approximate these encoders and extract hierarchical regularities, with potential benefits for explicit feature representations and higher-level concepts, though concrete optimization algorithms remain to be developed.

Abstract

Deep neural networks trained in an end-to-end manner are proven to be efficient in a wide range of machine learning tasks. However, there is one drawback of end-to-end learning: The learned features and information are implicitly represented in neural network parameters, which cannot be used as regularities, concepts or knowledge to explicitly represent the data probability distribution. To resolve this issue, we propose in this paper a new machine learning theory, which defines in mathematics what are regularities. Briefly, regularities are concise representations of the non-random features, or 'non-randomness' in the data probability distribution. Combining this with information theory, we claim that regularities can also be regarded as a small amount of information encoding a large amount of information. Our theory is based on spiking functions. That is, if a function can react to, or spike on specific data samples more frequently than random noise inputs, we say that such a function discovers non-randomness from the data distribution. Also, we say that the discovered non-randomness is encoded into regularities if the function is simple enough. Our theory also discusses applying multiple spiking functions to the same data distribution. In this process, we claim that the 'best' regularities, or the optimal spiking functions, are those who can capture the largest amount of information from the data distribution, and then encode the captured information in the most concise way. Theorems and hypotheses are provided to describe in mathematics what are 'best' regularities and optimal spiking functions. Finally, we propose a machine learning approach, which can potentially obtain the optimal spiking functions regarding the given dataset in practice.

Figures (11)

• Figure 1: True images versus random noise images. In the left, we present an upper part of Mona Lisa versus an image of white noise. In the right, we present an MNIST image for digit '7' MNIST_paper versus an image of the same size containing independent black-white pixels.
• Figure 2: The spiking regions of functions in $\mathbf{f}=(f_1,f_2,f_3)$ defined on the xy-plane. $\mathbf{S}_{f_1}$ is the blue circle in the front. $\mathbf{S}_{f_2}$ is the red circle but removing overlapping with $\mathbf{S}_{f_1}$. Then, $\mathbf{S}_{f_3}$ is the yellow circle after removing $\mathbf{S}_{f_1}$ and $\mathbf{S}_{f_2}$.
• Figure 3: Optimal encoders to several simple data distributions.
• Figure 4: Suppose the data space $\mathbf{S}\subset\mathbb{R}^2$, and suppose the second head of $F$ spikes on fixed random samples (points in the figure) in $\mathcal{D}'_{\text{fix},F}$. Then, the desired spiking region regarding the second head of $F$ should be the union of the circles, squares and polygons in the figure, converging to each random sample in $\mathcal{D}'_{\text{fix},F}$.
• Figure 5: Convolutional layer-wise regularity learning pipeline: Convolutional patches will be extracted from original images, which are used as input vectors to optimize bi-output functions. Then, the optimized bi-output functions are implemented back onto the original images to generate output tensors from their first heads. These output tensors are used as inputs to next-layer optimization.

Theorems & Definitions (4)

• proof
• proof
• proof
• proof