A Collision-Free Hot-Tier Extension for Engram-Style Conditional Memory: A Controlled Study of Training Dynamics

TL;DR

Engram-Nine tests whether removing high-frequency collisions via a Minimal Perfect Hash Function improves Engram-style memory during training under iso-parameter control. The study finds no statistically significant validation-loss benefit, suggesting collisions may provide implicit regularization and that precision alone is not the bottleneck. A route-stratified evaluation reveals a hot/cold flip during training and a gating preference that mismatches late-stage performance, pointing to gating credit assignment as a core mechanism. The findings imply practical guidance: retain collisions in data-scarce regimes and use the proposed diagnostics to understand memory dynamics before pursuing retrieval-precision enhancements.

Abstract

We investigate whether high-frequency key collisions are a primary bottleneck in Engram-style conditional memory. To isolate the effect of collisions, we introduce Engram-Nine, a collision-free hot-tier extension that maps the most frequent n-grams through a Minimal Perfect Hash Function (MPHF) while retaining the original multi-head hashed lookup as a cold tier. Under a strictly iso-parameter setup, the collision-free design does not consistently improve validation loss. Through route-stratified evaluation (decomposing per-token loss into hot/cold contributions), we uncover a consistent "hot-to-cold advantage flip" during training: hot (high-frequency) positions initially have lower loss, but cold positions eventually surpass them. Crucially, collision-free configurations flip earlier than collision-prone baselines, suggesting that collisions act as implicit regularization. We also identify a gating mismatch: the gate learns to favor hot positions early in training, but this preference persists even after the flip, assigning higher weights to positions with higher loss. Our findings suggest that improving lookup precision alone does not guarantee better training outcomes. The dominant limitation may lie in gating credit assignment rather than index accuracy, and collision-induced noise may provide beneficial regularization that should not be naively eliminated.

Figures (5)

• Figure 1: Engram-Nine two-tier retrieval architecture. Hot tier provides collision-free indexing via MPHF, cold tier retains the original multi-head hashing.
• Figure 2: Training dynamics of the Hot/Cold flip phenomenon. All configurations transition from early hot advantage (delta $>$ 0) to late cold advantage (delta $<$ 0). Nine configurations flip earlier than Hash configurations. (Hash-500K only preserved the final checkpoint, so it is not included in flip timing analysis.)
• Figure 3: Gate weight evolution during training (Nine-100/400K, seed0). Top: Raw $\alpha$ values show overall decline from $\sim$1.0 to $\sim$0.65. Middle: Gate preference $\Delta\alpha$ stays positive throughout, indicating persistent hot preference. Bottom: After iter 2500, cold achieves lower loss (negative region), but gate preference does not reverse—the preference "crystallized" in early training.
• Figure 4: Gate mismatch phenomenon. (a) High $\alpha$ bucket (0.8--1.0) has the highest average loss (5.1--5.3), low $\alpha$ bucket (0.2--0.4) has the lowest loss ($\approx$3.9). (b) Approximately 70% of the high $\alpha$ bucket are hot positions.
• Figure 5: Gating weight difference $\Delta\alpha = \alpha_{\text{hot}} - \alpha_{\text{cold}}$ across layers. Upper region (positive) indicates gate prefers hot tier, lower region (negative) indicates preference for cold tier. Hatched fill marks anomalous cold preference.