Shortcut Learning in In-Context Learning: A Survey

TL;DR

The paper addresses how shortcut learning emerges in In-Context Learning with large language models and its impact on robustness. It provides a structured taxonomy of shortcuts (instinctive vs acquired), analyzes their root causes in training data, demonstrations, and model scale, and reviews benchmarks and metrics for evaluation. It surveys data-, model-, and prompt-centric mitigation strategies, clarifying their applicability in the ICL setting and discussing limitations. The review highlights gaps in interpretability, unknown-shortcut scenarios, and benchmark design, and outlines directions for more robust evaluation, broader tasks, and understanding interactions between multiple shortcuts to guide future work in safe, reliable ICL systems.

Abstract

Shortcut learning refers to the phenomenon where models employ simple, non-robust decision rules in practical tasks, which hinders their generalization and robustness. With the rapid development of large language models (LLMs) in recent years, an increasing number of studies have shown the impact of shortcut learning on LLMs. This paper provides a novel perspective to review relevant research on shortcut learning in In-Context Learning (ICL). It conducts a detailed exploration of the types of shortcuts in ICL tasks, their causes, available benchmarks, and strategies for mitigating shortcuts. Based on corresponding observations, it summarizes the unresolved issues in existing research and attempts to outline the future research landscape of shortcut learning.

Figures (5)

• Figure 1: An example of shortcut learning in ICL.
• Figure 2: The organization of this survey.
• Figure 3: Examples of different instinctive shortcuts.
• Figure 5: Examples of different acquired shortcuts.
• Figure 7: Summary of shortcut mitigation methods.