Navigating High Dimensional Concept Space with Metalearning

TL;DR

This work probes the limits of gradient-based meta-learning for few-shot Boolean concept learning using a PCFG-controlled space that separately varies featural dimensionality $F$ and compositional depth $D$. By comparing Meta-SGD with SGD and analyzing inner-loop adaptation, loss-landscape roughness, and Hessian curvature, it shows that meta-learning is robust to compositional depth but struggles with high feature dimensionality, though longer adaptation and curvature-aware updates can mitigate some difficulties. The study reveals that Meta-SGD can dramatically shorten optimization paths and improve generalization in many regimes, especially when landscapes are rugged, but high-dimensional feature spaces still pose a challenge. These findings offer a nuanced view of when meta-learning helps in concept learning and emphasize the role of second-order dynamics and extended gradient adaptation in navigating complex loss landscapes.

Abstract

Rapidly learning abstract concepts from limited examples is a hallmark of human intelligence. This work investigates whether gradient-based meta-learning can equip neural networks with inductive biases for efficient few-shot acquisition of discrete concepts. I compare meta-learning methods against a supervised learning baseline on Boolean concepts (logical statements) generated by a probabilistic context-free grammar (PCFG). By systematically varying concept dimensionality (number of features) and recursive compositionality (depth of grammar recursion), I delineate between complexity regimes in which meta-learning robustly improves few-shot concept learning and regimes in which it does not. Meta-learners are much better able to handle compositional complexity than featural complexity. I highlight some reasons for this with a representational analysis of the weights of meta-learners and a loss landscape analysis demonstrating how featural complexity increases the roughness of loss trajectories, allowing curvature-aware optimization to be more effective than first-order methods. I find improvements in out-of-distribution generalization on complex concepts by increasing the number of adaptation steps in meta-SGD, where adaptation acts as a way of encouraging exploration of rougher loss basins. Overall, this work highlights the intricacies of learning compositional versus featural complexity in high dimensional concept spaces and provides a road to understanding the role of 2nd order methods and extended gradient adaptation in few-shot concept learning.

Figures (5)

• Figure 1: The PCFG parse trees of concepts with increasing complexity. Here compositional depth is visualized as the depth of the parse tree on the vertical axis, feature dimensionality is visualized as the width of the parse tree on the horizontal axis. Examples show how PCFG-generated concepts scale from simple to complex logical structures. Left: Simple concept with 2 features and depth 3. Center: Medium complexity with 3 features and depth 4. Right: Complex concept with 5 features and depth 5. Neural networks see only the bit-string input of features and ideally learn to infer the logical structure of the underlying concept over successive trials.
• Figure 2: First order meta-SGD (blue lines) versus second order meta-SGD (green lines) and vanilla SGD (red line) performance over increasingly complex Boolean concepts. Featural complexity (number of literals) increases along (rows) and concept depths (columns) over normalized training episodes.
• Figure 3: Meta-SGD and SGD operate on the same concept loss landscapes (determined by task structure and architecture), but meta-learning learns more efficient navigation strategies (shorter paths to the solution point - the bottom-most point in each loss 'basin'). Top row: 2D loss landscapes for simple, medium, and complex Boolean concepts show identical topology regardless of optimization method. Middle row: 3D visualizations reveal the terrain both algorithms must navigate, with complexity-dependent roughness. Note: due to computational intractability, loss landscapes are local approximations.
• Figure 4: Training samples to reach 60% validation accuracy (log scale). While results are mixed and vanilla SGD never achieves the floor accuracy of 60%, there are intriguing early trends for first versus second order Meta-SGD. FOr example, while first order Meta-SGD outperforms in high depth settings with a low number of features, second order greatly outperforms when increasing featural complexity. The 32-feature case (bottom panel) has only data for first order Meta-SGD with k=10 steps because it was the only method to generalize to above 60 percent accuracy.
• Figure 5: K=1 vs K=10 Adaptation Steps Scale with Landscape Complexity. Top:Accuracy improvements from K=1 to K=10 scale predictably with landscape complexity, showing modest gains for smooth landscapes but substantial for rough. Bottom: Sample efficiency analysis reveals that additional adaptation steps provide increasingly large benefits as optimization landscapes become rougher.