Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Measuring Sharpness in Grokking

Jack Miller, Patrick Gleeson, Charles O'Neill, Thang Bui, Noam Levi

TL;DR

This work tackles the problem of quantifying the sharpness of grokking transitions by fitting a shared S-shaped function to both training and generalization accuracy curves, extracting jump time $t^*$ and sharpness $s$, and deriving the relative grokking gap $m$ and relative sharpness $R_{\mathrm{rel}}$. The authors propose two measures, $R_{\mathrm{rel}}$ and $R_{\mathrm{abs}}$, and validate the approach in two settings: Levi's linear student–teacher model with closed-form dynamics and a two-layer MLP with concealed parity using spurious features. They find that, across both settings, larger relative gaps $m$ tend to accompany smaller sharpness values, indicating softer transitions as grokking becomes more pronounced. The results offer a robust, comparative framework for analyzing grokking across architectures and tasks and point to further theoretical and empirical work to understand the mechanisms governing sharpness.

Abstract

Neural networks sometimes exhibit grokking, a phenomenon where perfect or near-perfect performance is achieved on a validation set well after the same performance has been obtained on the corresponding training set. In this workshop paper, we introduce a robust technique for measuring grokking, based on fitting an appropriate functional form. We then use this to investigate the sharpness of transitions in training and validation accuracy under two settings. The first setting is the theoretical framework developed by Levi et al. (2023) where closed form expressions are readily accessible. The second setting is a two-layer MLP trained to predict the parity of bits, with grokking induced by the concealment strategy of Miller et al. (2023). We find that trends between relative grokking gap and grokking sharpness are similar in both settings when using absolute and relative measures of sharpness. Reflecting on this, we make progress toward explaining some trends and identify the need for further study to untangle the various mechanisms which influence the sharpness of grokking.

Measuring Sharpness in Grokking

TL;DR

This work tackles the problem of quantifying the sharpness of grokking transitions by fitting a shared S-shaped function to both training and generalization accuracy curves, extracting jump time and sharpness , and deriving the relative grokking gap and relative sharpness . The authors propose two measures, and , and validate the approach in two settings: Levi's linear student–teacher model with closed-form dynamics and a two-layer MLP with concealed parity using spurious features. They find that, across both settings, larger relative gaps tend to accompany smaller sharpness values, indicating softer transitions as grokking becomes more pronounced. The results offer a robust, comparative framework for analyzing grokking across architectures and tasks and point to further theoretical and empirical work to understand the mechanisms governing sharpness.

Abstract

Neural networks sometimes exhibit grokking, a phenomenon where perfect or near-perfect performance is achieved on a validation set well after the same performance has been obtained on the corresponding training set. In this workshop paper, we introduce a robust technique for measuring grokking, based on fitting an appropriate functional form. We then use this to investigate the sharpness of transitions in training and validation accuracy under two settings. The first setting is the theoretical framework developed by Levi et al. (2023) where closed form expressions are readily accessible. The second setting is a two-layer MLP trained to predict the parity of bits, with grokking induced by the concealment strategy of Miller et al. (2023). We find that trends between relative grokking gap and grokking sharpness are similar in both settings when using absolute and relative measures of sharpness. Reflecting on this, we make progress toward explaining some trends and identify the need for further study to untangle the various mechanisms which influence the sharpness of grokking.
Paper Structure (8 sections, 13 equations, 6 figures)

This paper contains 8 sections, 13 equations, 6 figures.

Table of Contents

  1. Introduction
  2. Measuring Grokking
  3. Experiments
  4. Linear Student Teacher Setup
  5. MLP with Concealed Parity Prediction
  6. Discussion
  7. Empirical Evidence for Error Function Fits
  8. Log-Log Plots for Experiments

Figures (6)

  • Figure 1: Relationship between relative grokking gap and both measures under the linear framework. For both measures we took $\lambda \in [1.003, 1.1]$, $\epsilon = 10^{-10}$ and $\eta_0 = 0.01$. Note that lines simply connect the data points and do not represent a trend. See \ref{['appendix:log-log-plots']} for linear trends in log-log space.
  • Figure 2: Relationship between relative grokking gap and $R_\mathrm{rel}$ and $R_\mathrm{abs}$. Each dot represents a training run.
  • Figure 3: Visual evidence of \ref{['eqn:erf-form']} fit in the linear estimators setting. Note for numerical reasons, the domain for each $\lambda$ is distinct.
  • Figure 4: Visual evidence of \ref{['eqn:erf-form']} fit in the MLP setting. Note that input size refers to the changing size of examples resulting from the addition of further spurious dimensions.
  • Figure 5: Same setting as in \ref{['fig:empirical-on-mlp']} but with a log-log scale and linear trend.
  • ...and 1 more figures