Randomly Weighted Neuromodulation in Neural Networks Facilitates Learning of Manifolds Common Across Tasks

TL;DR

The paper introduces Task-specific Geometric Sensitive Hashing (T-GSH) to formalize how neural representations reflect task manifolds across sequential supervised tasks. It shows that Configurable Random Weight Networks (CRWNs), which use fixed random bases plus global/local neuromodulation, realize a T-GSH mapping by learning task-specific modulations $B^{t}$ while keeping a shared random base $\mathbf{R}$. Through RotationMNIST, ShiftMNIST, and AugmentMNIST experiments, the authors demonstrate that task-context vectors encode meaningful task relationships and that the manifold structure governing tasks is recoverable from representations. This work provides a geometry-driven, neuromodulation-inspired perspective on continual learning with potential theoretical and practical implications for cross-task representation sharing and transfer.

Abstract

Geometric Sensitive Hashing functions, a family of Local Sensitive Hashing functions, are neural network models that learn class-specific manifold geometry in supervised learning. However, given a set of supervised learning tasks, understanding the manifold geometries that can represent each task and the kinds of relationships between the tasks based on them has received little attention. We explore a formalization of this question by considering a generative process where each task is associated with a high-dimensional manifold, which can be done in brain-like models with neuromodulatory systems. Following this formulation, we define \emph{Task-specific Geometric Sensitive Hashing~(T-GSH)} and show that a randomly weighted neural network with a neuromodulation system can realize this function.

Figures (7)

• Figure 1: (a) Each task $t$ consists of a set of points of classes on a simple manifold. Furthermore, each point in the task can be characterized by three latent parameters: $\boldsymbol{\gamma}^{t}, \boldsymbol{\delta}, \boldsymbol{\theta}^{t}$. Notably, we consider a task-specific manifold broader than the class-level one in dikkala2021manifold. (b) Any set of points on the same task manifold map to (approximately) the same representation (i.e., the penultimate layer feature map), while a set of points from different task manifolds go to far away representations.
• Figure 2: Experiment results on RotationMNIST.
• Figure 3: Empirical evidence of approximate data generating process with our defined invertibility assumption on RotationMNIST. (Top Row) Reconstructed samples of digit "3". (Bottom Row) Reconstructed samples of digit "1". (Left Column) Reconstructed samples of digits from the task manifold $M^{T1}$, which is $0$° rotation. (Middle Column) the samples of digits from the task manifold $M^{T5}$, which is $40$ ° counterclockwise rotation. (Right Column) the samples of digits from the task manifold $M^{T10}$, which is $90$ ° counterclockwise rotation.
• Figure 4: ShiftMNIST-1px. (\ref{['fig:shiftmnist_task_01']}) The difference between the datasets of the two adjacent tasks, $i$ and $i+1$, is that the images of task $i+1$ are the images of task $i$ shifted a single pixel to the right. Thus, the total number of tasks to learn is 28. (\ref{['fig:shiftmnist_cosinesim_01']}) Cosine similarity between the context vectors for each task before (left) and after training (right).
• Figure 5: ShiftMNIST-2px. (\ref{['fig:shiftmnist_task_02']}) The difference between the datasets of the two adjacent tasks, $i$ and $i+1$, is that the images of task $i+1$ are the images of task $i$ shifted two pixels to the right. Thus, the total number of tasks to learn is 14. (\ref{['fig:shiftmnist_cosinesim_02']}) Cosine similarity between the context vectors for each task before (left) and after training (right).

Theorems & Definitions (1)

• Definition 1: Task-specific Geometric Sensitive Hashing (T-GSH)