Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks

TL;DR

This work investigates whether phonemic, lexical semantic, and syntactic representations naturally emerge in self-supervised neural networks during training. By extending a linear Structural Probe to multiple linguistic levels and evaluating both speech and text models, it reveals that activations organize into subspaces whose geometry matches articulatory, semantic, and syntactic relations, with emergence proceeding in a phoneme → lexical semantics → syntax order. It shows that larger models and speech-informed data boost the fidelity and speed of these emergent structures, but also highlights a substantial data-efficiency gap relative to human language acquisition. The findings provide a unified, geometry-based framework for understanding language representations in neural networks and propose a path toward linking computational mechanisms in artificial systems with developmental trajectories observed in humans, while pointing to future work in brain alignment and improved training paradigms.

Abstract

During language acquisition, children successively learn to categorize phonemes, identify words, and combine them with syntax to form new meaning. While the development of this behavior is well characterized, we still lack a unifying computational framework to explain its underlying neural representations. Here, we investigate whether and when phonemic, lexical, and syntactic representations emerge in the activations of artificial neural networks during their training. Our results show that both speech- and text-based models follow a sequence of learning stages: during training, their neural activations successively build subspaces, where the geometry of the neural activations represents phonemic, lexical, and syntactic structure. While this developmental trajectory qualitatively relates to children's, it is quantitatively different: These algorithms indeed require two to four orders of magnitude more data for these neural representations to emerge. Together, these results show conditions under which major stages of language acquisition spontaneously emerge, and hence delineate a promising path to understand the computations underpinning language acquisition.

Figures (15)

• Figure 1: Linear probing recovers phonemic, lexical semantic, and syntactic structure from the activation spaces of neural networks trained via Self-Supervised Learning. The networks' neural activations live on a high-dimensional space in which different linear subspaces coexist. Different linear probes recover linear subspaces which represent phonemic, lexical semantic and syntactic structures, as long postulated by linguistic theories.
• Figure 2: Phoneme: structural probes recover the geometry of phonemic articulation.A. Phoneme representations from a 2D structural probe trained to match pairwise articulatory distances among vowels. Responses to the same phoneme are colored identically. B. Canonical vowel trapezium indicating tongue height (vertical) and backness (horizontal). C. Layer-wise phonemic score for Wav2Vec2: probes trained on the final pretraining checkpoint peak near 80% relative depth and consistently outperform probes trained on the initial (untrained) checkpoint. D. Effect of model and data: English pretraining and larger model capacity both yield higher phonemic scores. E. Phonemic structure emerges gradually with pretraining, with scores increasing with the number of pretraining steps; the data required for Wav2Vec 2.0 to acquire robust phonemic structure far exceeds typical estimates of children’s linguistic input.
• Figure 3: Structural probes recover the geometry of lexical semantic graphs A: 2d projection of text model (pythia-1.4B) response to all words in the mammal subgraph. B: Example WordNet subgraph, displaying synsets of the mammal subgraph with more than 50 hyponyms. c: Semantic score (for the graph composed of all nouns) for a large text (Llama-13B), text (pythia-1.4B), audio (Wav2vec 2.0-94M base), and random (Wav2Vec 2.0-94M base) models. D: Semantic scores for all models as a function of the model size. E: Semantic score as a function of the quantity of pertaining for the text (pythia-1.4B) and audio (Wav2vec 2.0-94M base) models.
• Figure 4: Structural probes recover the geometry of syntactic trees A. In each left subpanel, we plot an example of a 2d probe projection of the model activations, with trees reconstructed through the minimum spanning tree algorithm. On the right subpanel, we plot the gold (yellow) and predicted (black). B. Example of sentences and their syntactic trees. C. Syntactic scores of a large text (Llama-13B), a text (pythia-1B), speech (Wav2vec 2.0-94M base) and random (Wav2vec 2.0-94M base) models for each transformer layer. D. Highest syntactic score across layer for each model. E. Emergence of the syntactic abilities during pretraining, which includes several repetitions of the same dataset for the audio model.
• Figure 5: Linguistic structures are acquired in sequential order A, B and C: 2d visualizations of the phonemic and semantic space along with one example syntactic structure for 3 pertaining steps. D: Left: Emergence of semantic, syntactic, and phonemic score for a Wav2vec 2.0-94M base model (dots), along with a parametric fit (line) E: relative scores of the parametric fit (min to 0 and max to 1), demonstrating the successive emergence of phonemic and lexical-syntactic structures. Checkpoints used for the top sub-panel are highlighted with lines. F: average amount of words (tokens) heard by a child or used by models. Note that for the audio dataset, many epochs of optimization over the same dataset take place.