Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Intrinsic Task Symmetry Drives Generalization in Algorithmic Tasks

Hyeonbin Hwang, Yeachan Park

TL;DR

It is shown that generalization emerges during the symmetry acquisition phase, after which representations reorganize into a structured, task-aligned geometry.

Abstract

Grokking, the sudden transition from memorization to generalization, is characterized by the emergence of low-dimensional representations, yet the mechanism underlying this organization remains elusive. We propose that intrinsic task symmetries primarily drive grokking and shape the geometry of the model's representation space. We identify a consistent three-stage training dynamic underlying grokking: (i) memorization, (ii) symmetry acquisition, and (iii) geometric organization. We show that generalization emerges during the symmetry acquisition phase, after which representations reorganize into a structured, task-aligned geometry. We validate this symmetry-driven account across diverse algorithmic domains, including algebraic, structural, and relational reasoning tasks. Building on these findings, we introduce a symmetry-based diagnostic that anticipates the onset of generalization and propose strategies to accelerate it. Together, our results establish intrinsic symmetry as the key factor enabling neural networks to move beyond memorization and achieve robust algorithmic reasoning.

Intrinsic Task Symmetry Drives Generalization in Algorithmic Tasks

TL;DR

It is shown that generalization emerges during the symmetry acquisition phase, after which representations reorganize into a structured, task-aligned geometry.

Abstract

Grokking, the sudden transition from memorization to generalization, is characterized by the emergence of low-dimensional representations, yet the mechanism underlying this organization remains elusive. We propose that intrinsic task symmetries primarily drive grokking and shape the geometry of the model's representation space. We identify a consistent three-stage training dynamic underlying grokking: (i) memorization, (ii) symmetry acquisition, and (iii) geometric organization. We show that generalization emerges during the symmetry acquisition phase, after which representations reorganize into a structured, task-aligned geometry. We validate this symmetry-driven account across diverse algorithmic domains, including algebraic, structural, and relational reasoning tasks. Building on these findings, we introduce a symmetry-based diagnostic that anticipates the onset of generalization and propose strategies to accelerate it. Together, our results establish intrinsic symmetry as the key factor enabling neural networks to move beyond memorization and achieve robust algorithmic reasoning.
Paper Structure (97 sections, 9 theorems, 26 equations, 55 figures, 2 tables)

This paper contains 97 sections, 9 theorems, 26 equations, 55 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. General Setup
  4. Modular Arithmetic
  5. Graph Metric Completion
  6. Comparison
  7. Motivation: Structured Geometry in Embedding Space
  8. Intrinsic Symmetries in Algorithmic Tasks
  9. Moodular Arithmetics
  10. Graph Metric Completion
  11. Comparison
  12. Symmetry-Driven Generalization
  13. Three-Stage Training Dynamics
  14. Symmetry-Based Criterion
  15. Discussion
  16. ...and 82 more sections

Key Result

Proposition 7.1

Let $\mathcal{M}$ be an one-dimensional compact abelian topological group and $\mathbb{Z}_p$ is continuously embedded in $\mathcal{M}$. Then $\mathcal{M}$ is isomorphic to $S^1$ embedded in high-dimensional torus $\mathbb{T}^D$ (closed helix) Hence $\Phi(\mathbb{Z}_p) \hookrightarrow \Phi(\mathcal{M})$ is also confined in the closed helix.

Figures (55)

  • Figure 1: Modular Arithmetics Embeddings
  • Figure 2: Graph Metric Completion and Comparison Embeddings.
  • Figure 3: Type of Graphs.
  • Figure 4: Full PCA visualization of modular addition embeddings.
  • Figure 5: Symmetric patterns within a dataset lead the network to learn intrinsic symmetry.
  • ...and 50 more figures

Theorems & Definitions (13)

  • Proposition 7.1
  • Proposition 7.2
  • Proposition 7.3
  • Definition 8.1: Pontryagin dual
  • Lemma 8.2
  • Lemma 8.3
  • Theorem 8.4: Pontryagin duality pontrjagin1934theorymorris1977pontryaginhewitt2013abstract
  • Proposition
  • proof
  • Proposition
  • ...and 3 more