Training Memory in Deep Neural Networks: Mechanisms, Evidence, and Measurement Gaps

TL;DR

This survey tackles the nontrivial memory embedded in modern neural network training, arguing that updates depend not only on current state but on optimizer moments, data-order policies, and the trajectory through parameter space. It introduces a three-axis taxonomy (Source, Lifetime, Visibility), portable perturbation primitives, and a concrete reporting checklist to enable causal, uncertainty-aware attribution of how much training history matters. By unifying optimizer, sampler, and path effects—and by proposing seed-paired interventions and function-space readouts—the paper lays out a principled protocol for measuring training memory across models, data, and regimes without prescribing a single diagnostic. The work also outlines practical desiderata, benchmarks, and cross-domain insights from continual learning, curriculum design, data selection, RL replay, and federated optimization to advance memory-aware diagnostics with reproducible uncertainty estimates. Overall, it seeks to turn training memory from folklore into a cumulative science with transparent perturbations, auditable artifacts, and robust evaluation across modalities.

Abstract

Modern deep-learning training is not memoryless. Updates depend on optimizer moments and averaging, data-order policies (random reshuffling vs with-replacement, staged augmentations and replay), the nonconvex path, and auxiliary state (teacher EMA/SWA, contrastive queues, BatchNorm statistics). This survey organizes mechanisms by source, lifetime, and visibility. It introduces seed-paired, function-space causal estimands; portable perturbation primitives (carry/reset of momentum/Adam/EMA/BN, order-window swaps, queue/teacher tweaks); and a reporting checklist with audit artifacts (order hashes, buffer/BN checksums, RNG contracts). The conclusion is a protocol for portable, causal, uncertainty-aware measurement that attributes how much training history matters across models, data, and regimes.

Figures (5)

• Figure 1: Schematic of the training loop with explicit and implicit state. Boxes: weights $\theta_t$, optimizer buffers (e.g., momentum/Adam), sampler RNG/order, external queues/memory banks, and teacher/EMA. Arrows indicate what carries across steps and phases; coloring encodes visibility/resetability.
• Figure 2: Typical memory lifetimes by source (S1–S5), from steps $\rightarrow$ epochs $\rightarrow$ phases $\rightarrow$ tasks/rounds. Each band shows a suggested intervention window $W$ suited to that source: step-scale state (e.g., momentum/Adam, EMA) use $W\!\approx\!1$–$2$ half-lives; epoch-scale order use one full reshuffled epoch or a fixed minibatch window under with-replacement sampling; phase-scale effects probe the first $k$ epochs post-boundary; external queues use the queue turnover (queue length divided by enqueue rate); short AB$\neq$BA order swaps at boundaries expose non-commutativity.
• Figure 3: Resetability $\times$ auditability matrix with examples: ideal (visible, resetable, auditable) vs. visible-but-hard-to-reset, resetable-but-unlogged, and implicit/path dependence.
• Figure 4: Federated learning rounds with server-side memory: client local updates aggregate into server accumulators (e.g., momentum $m_t$, second moments $v_t$), which persist across rounds; participation masks and sampling policy contribute additional round-scale memory.
• Figure 5: Branch-and-hold design at time $t$: fork runs that differ only in a targeted source (e.g., carry vs. reset optimizer state, or order window swap) for a window $W$, then continue to horizon $T$. Report paired seed effect sizes $\Delta M$ in function space with CIs.