Learning Hierarchically Structured Concepts

TL;DR

This work develops a theory for how hierarchically structured concepts can be represented, learned, and recognized in brain-inspired, feed-forward spiking neural networks. It formalizes concept hierarchies as trees and embeds them with a corresponding layered SNN model that uses Oja's learning rule and Winner-Take-All sub-networks to achieve robust recognition and bottom-up learning. The authors provide both noise-free and noisy-learning algorithms with convergence guarantees, and establish a preliminary lower bound showing depth is tied to hierarchical depth. They also outline extensions to more flexible hierarchies, noisy scenarios, and network variations, highlighting future directions toward a broader theory of hierarchical concept representations in neural networks. Overall, the paper contributes formal definitions, algorithms, and theoretical limits that connect hierarchical structure with learnability in biologically plausible neural systems, potentially informing both neuroscience and machine learning foundations.

Abstract

We study the question of how concepts that have structure get represented in the brain. Specifically, we introduce a model for hierarchically structured concepts and we show how a biologically plausible neural network can recognize these concepts, and how it can learn them in the first place. Our main goal is to introduce a general framework for these tasks and prove formally how both (recognition and learning) can be achieved. We show that both tasks can be accomplished even in presence of noise. For learning, we analyze Oja's rule formally, a well-known biologically-plausible rule for adjusting the weights of synapses. We complement the learning results with lower bounds asserting that, in order to recognize concepts of a certain hierarchical depth, neural networks must have a corresponding number of layers.

Figures (3)

• Figure 1: The leftmost figure shows the concept human, which consists of two sub-concepts, and so on. The second figure shows a network that has "learned" the concept "human" in the sense that, when the neurons representing the basic parts eyes, mouth, arms, legs are excited, then exactly one neuron $u$ on the top layer will fire. Neuron $u$ should also fire when "most" of the basic parts are excited, and $u$ should not fire when few of the basic parts are excited. For example, the painting "Girl with a Mandolin" by Picassoshould cause $u$ to fire despite the lack of a mouth and legs. The network accomplishes this by strengthening relevant synapses (bold edges) and weakening others (thin edges).
• Figure 2: This example illustrates the $supported_r(B)$ definition, with $k = 3$ and $r = \frac{2}{3}$. We depict just a single level $2$ concept $c$ with children $c_1, c_2, c_3$ and grandchildren $c_{1,1}, c_{1,2}, c_{1,3}, c_{2,1}, c_{2,2}, c_{2,3}, c_{3,1}, c_{3,2}, c_{3,3}$. The set $B$ consists of concepts $c_{1,1}$, $c_{1,2}$, $c_{3,1}, c_{3,3}$ plus an "extra" concept $c_{4,0}$ that is not a descendant of $c$. Then $B_0 = \{c_{1,1}, c_{1,2}, c_{3,1}, c_{3,3}\}$, $B_1 = \{c_1, c_3\}$, and $B_2 = \{c\}$.
• Figure 3: The figure depicts the general structure of a feed-forward network.

Theorems & Definitions (38)

• Definition 1: Supported
• Lemma 1
• Definition 2: Presented
• Definition 3: Robust recognition problem
• Lemma 2
• proof
• Definition 4: Showing a concept
• Definition 5: $\sigma$-bottom-up training schedule
• Definition 6: $(r_1,r_2,\sigma)$-learning
• Theorem 5.1