Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks

TL;DR

This work examines the interplay between sequence transformation blocks and regressive performance in-context in GPT-2/LLaMa hybrid and LLaMa/Mamba hybrid models - examining the interplay between sequence transformation blocks and regressive performance in-context.

Abstract

In-Context Learning (ICL) is a phenomenon where task learning occurs through a prompt sequence without the necessity of parameter updates. ICL in Multi-Headed Attention (MHA) with absolute positional embedding has been the focus of more study than other sequence model varieties. We examine implications of architectural differences between GPT-2 and LLaMa as well as LlaMa and Mamba. We extend work done by Garg et al. (2022) and Park et al. (2024) to GPT-2/LLaMa hybrid and LLaMa/Mamba hybrid models - examining the interplay between sequence transformation blocks and regressive performance in-context. We note that certain architectural changes cause degraded training efficiency/ICL accuracy by converging to suboptimal predictors or converging slower. We also find certain hybrids showing optimistic performance improvements, informing potential future ICL-focused architecture modifications. Additionally, we propose the "ICL regression score", a scalar metric describing a model's whole performance on a specific task. Compute limitations impose restrictions on our architecture-space, training duration, number of training runs, function class complexity, and benchmark complexity. To foster reproducible and extensible research, we provide a typed, modular, and extensible Python package on which we run all experiments.

Figures (21)

• Figure 1: Visual aid for our explored hybrid models in tabular and graphical format.
• Figure 2: Predictors and conditions for computation and interpretation of ICL regression score.
• Figure 3: A flow map of the various objects in our codebase. To conduct similar ICL experiments, users only need to provide a YAML configuration file. For training, this configuration specifies the function class, model type, and curriculum. For evaluation, this configuration specifies the function class, model, and metric.
• Figure 4: Squared error profiles that do not exhibit near-optimal behavior. Shaded regions are 99% confidence intervals.
• Figure 5: Detailing plots to showcase GPT-2 RMS SwiGLU (model 1.4) learning a more general but sub-optimal regression scheme when trained on Sparse Linear Regression. Shaded regions are 99% confidence intervals.