Schema for In-Context Learning

TL;DR

It is demonstrated that a broad range of large language models lack the capacity to form and utilize internal schema-based learning representations implicitly, but instead benefit significantly from explicit schema-based scaffolding, and paves a new path for enhancing human-like reasoning in LLMs.

Abstract

In-Context Learning (ICL) enables transformer-based language models to adapt to new tasks by conditioning on demonstration examples. However, traditional example-driven in-context learning lacks explicit modules for knowledge retrieval and transfer at the abstraction level. Inspired by cognitive science, specifically schema theory, which holds that humans interpret new information by activating pre-existing mental frameworks (schemas) to structure understanding, we introduce SCHEMA ACTIVATED IN CONTEXT LEARNING (SA-ICL). This framework extracts the representation of the building blocks of cognition for the reasoning process instilled from prior examples, creating an abstracted schema, a lightweight, structured template of key inferential steps and their relationships, which is then used to augment a model's reasoning process when presented with a novel question. We demonstrate that a broad range of large language models (LLMs) lack the capacity to form and utilize internal schema-based learning representations implicitly, but instead benefit significantly from explicit schema-based scaffolding. Across chemistry and physics questions from the GPQA dataset, our experiments show that SA-ICL consistently boosts performance, up to 36.19 percent, when the single demonstration example is of high quality, which simultaneously reduces reliance on the number of demonstrations and enhances interpretability. SCHEMA ACTIVATED IN CONTEXT LEARNING not only bridges disparate ICL strategies ranging from pattern priming to Chain-of-Thought prompting, but also paves a new path for enhancing human-like reasoning in LLMs.

Figures (7)

• Figure 1: SA-ICL applied to a coin-weighing optimization problem. The framework demonstrates five sequential stages: (i) generates problem representation to form an initial schema recognizing this as a mathematical optimization and partitioning challenge, (ii) retrieves a similar prior schema to identify relevant properties of the question, (iii) gathers a subset of past influential episodic examples that are relevant to solve the target problem, (iv) performs schema activation and integrates retrieved knowledge to develop optimal mathematical strategies adapted to solve the target problem, and (v) conditions the target problem to the adapted schema and utilizes schema-guided inference to eventually conclude that the maximum number of guaranteed real coins is 142.
• Figure 2: The performances of six LLMs on Chemistry and Physics questions using SA-ICL and example-driven One-Shot prompting, with two retrieval methods \ref{['sec:similarity_how']}. The black dotted line represents the zero-shot performance.
• Figure D.1: The heatmaps for knowledge density for GPQA-Chemistry and GPQA-Physics
• Figure G.2: Performance of GPT-5 on the chemistry, physics, biology multi-choice questions from the Humanity's Last Exam (HLE) benchmark dataset phan2025humanity (N=191, we filtered out questions with images), when provided with the Essentially Same examples. We generated the synthetic data as well using the same procedures described in \ref{['app:synthetic_data_generation']}.
• Figure H.3: We conducted experiments in chemistry and physics using the GPQA dataset, incorporating synthetic examples with predefined levels of prior knowledge similarity \ref{['sec:similarity_how']}. Across all knowledge levels, incorporating schema-based examples consistently yielded notable performance improvements. This effect was particularly pronounced when the prior knowledge examples exhibited high similarity to the target questions. The number contained within each bar represents the number of experiments run. The numbers in the bars represent number of questions iterated.