In-context Learning with Retrieved Demonstrations for Language Models: A Survey
Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, Mehran Kazemi
TL;DR
This survey analyzes the rise of retrieval-based in-context learning (RetICL) for large language models, outlining how dynamically retrieved demonstrations tailored to each query improve performance over fixed examples. It dissects design choices across retrieval objectives, strategies, and corpora, and contrasts off-the-shelf versus fine-tuned retrievers, including training signals and loss functions. The authors categorize applications into NLU, reasoning, knowledge-based QA, and text generation, and discuss practical considerations, limitations, and directions such as active retrieval, cross-domain transfer, and multimodal extensions. Overall, RetICL emerges as a scalable, bias-reducing paradigm that enhances few-shot learning by leveraging targeted demonstrations from diverse data sources. The report highlights open questions about why similarity and diversity matter and emphasizes methodological advances in retriever training and evaluation to broaden RetICL’s applicability.
Abstract
Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model's ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.