lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning

TL;DR

The paper tackles how training data relationships influence generalisation in neural networks by deriving lpNTK, a label-aware extension of the NTK that captures sample interactions during learning. Through a first-order Taylor analysis of SGD updates, lpNTK incorporates label information to define a scalar similarity between labelled samples and proves it asymptotically approaches the empirical NTK under width growth. This framework unifies explanations for learning difficulty and forgetting events via three relation types—interchangeable, unrelated, and contradictory—and demonstrates practical value by showing that removing redundant samples identified by lpNTK does not hurt, and can even improve, generalisation on MNIST and CIFAR-10. The authors further show that selective pruning guided by lpNTK clusters can counteract dataset biases toward interchangeable samples, aligning with broader data-centric learning findings and offering a data-pruning strategy with potential general impact for coreset design and model robustness. Overall, lpNTK provides a principled, label-aware lens on learning dynamics and data selection, revealing that careful data pruning based on sample interactions can achieve better generalisation with less data.

Abstract

Although much research has been done on proposing new models or loss functions to improve the generalisation of artificial neural networks (ANNs), less attention has been directed to the impact of the training data on generalisation. In this work, we start from approximating the interaction between samples, i.e. how learning one sample would modify the model's prediction on other samples. Through analysing the terms involved in weight updates in supervised learning, we find that labels influence the interaction between samples. Therefore, we propose the labelled pseudo Neural Tangent Kernel (lpNTK) which takes label information into consideration when measuring the interactions between samples. We first prove that lpNTK asymptotically converges to the empirical neural tangent kernel in terms of the Frobenius norm under certain assumptions. Secondly, we illustrate how lpNTK helps to understand learning phenomena identified in previous work, specifically the learning difficulty of samples and forgetting events during learning. Moreover, we also show that using lpNTK to identify and remove poisoning training samples does not hurt the generalisation performance of ANNs.

Figures (18)

• Figure 1: Correlation between the learning difficulty of samples trained on subsets of varying sizes. For each subfigure, the x-value is the difficulty of samples trained on the universal set ($4096$), while y-axis is the difficulty trained on the contrast setting ($1024$, $256$, etc.). Smaller Pearson correlation coefficient $\rho$ means less correlation between x and y values. The title of each panel is the settings we compare, e.g. "4096 vs 1" means that we plot the learning difficulty of the same sample on the universal set (of size $4096$) against its learnability in a dataset containing just itself.
• Figure 2: Learning difficulty of target samples, i.e. centroids of FPC head clusters, when the samples in the training set are interchangeable (red line), non-interchangeable (purple line), or medium interchangeable (yellow line). The plots show that the learning difficulty of target samples can be controlled through adding samples with different relationships to them into the training sets.
• Figure 3: Visualisation of three possible relationships between data samples. Each vector represents an update to parameters of models based on a back-propagation from a sample e.g. ${\color{orange} {\bm{x}}_o}$ or ${\color{blue} {\bm{x}}_u}$ and their corresponding labels. Formal definitions are given below.
• Figure 4: Visualisation of the correlation between learning difficulty on MNIST where $N=4096, X\in\{1,4,16,64,256,1024\}$, learning rate is set to $0.1$, and batch size is $128$, i.e. setting \ref{['appexp:mnist_0.1']}. Note that we plot only the correlation between $1000$ samples in order to keep the plots readable.
• Figure 5: Visualisation of the correlation between learning difficulty on MNIST where $N=4096, X\in\{1,4,16,64,256,1024\}$, learning rate is set to $0.001$, and batch size is $256$, i.e. setting \ref{['appexp:mnist_0.001']}. Note that we plot only the correlation between $1000$ samples in order to keep the plots readable.

Theorems & Definitions (1)

• Theorem 1