Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning

TL;DR

This work addresses how exemplar selection for in-context learning can benefit from both semantic similarity and the structural logic of problem-solving. It introduces Reasoning Graph Enhanced Exemplar Retrieval (RGER), which builds reasoning graphs from intermediate steps and uses graph kernels to measure structural similarity for re-ranking candidate exemplars before prompting an LLM. By combining a learned retriever with graph-based re-ranking and two graph-construction pipelines, RGER achieves superior performance on math and logic reasoning benchmarks across multiple backends and demonstrates robustness to different model capabilities. The approach advances ICL by explicitly aligning problem-solving trajectories between exemplars and queries, enabling more reliable and explainable in-context reasoning with practical impact for complex tasks.

Abstract

Large language models (LLMs) have exhibited remarkable few-shot learning capabilities and unified the paradigm of NLP tasks through the in-context learning (ICL) technique. Despite the success of ICL, the quality of the exemplar demonstrations can significantly influence the LLM's performance. Existing exemplar selection methods mainly focus on the semantic similarity between queries and candidate exemplars. On the other hand, the logical connections between reasoning steps can be beneficial to depict the problem-solving process as well. In this paper, we proposes a novel method named Reasoning Graph-enhanced Exemplar Retrieval (RGER). RGER first quires LLM to generate an initial response, then expresses intermediate problem-solving steps to a graph structure. After that, it employs graph kernel to select exemplars with semantic and structural similarity. Extensive experiments demonstrate the structural relationship is helpful to the alignment of queries and candidate exemplars. The efficacy of RGER on math and logit reasoning tasks showcases its superiority over state-of-the-art retrieval-based approaches. Our code is released at https://github.com/Yukang-Lin/RGER.

Figures (6)

• Figure 1: Comparison between semantically similar(top) and structurally similar(bottom) for in-context demonstrations. (Top)The exemplar with semantic similarity focuses more on detailed information like the word "bakery" and the synonyms "muffins", "pastries", "cupcakes", and so on. (Bottom)The exemplar emphasizes the structural similarity between the exemplar and the question, which likely leads to a correct response.
• Figure 2: The pipeline overview of the proposed method. It consists of two parts: two-time queries (top) and retrieval process RGER (bottom).
• Figure 3: (Top) The approach prompts language models to generate a deductive reasoning form response, then extract the pathway to build a graph. (Bottom) The approach uses language models to extract equations from a response and then build a graph by Algorithm \ref{['alg:algorithm']}.
• Figure 4: Comparison of two re-ranking methods, LoRe and Graph Similarity, under varying response quality levels.
• Figure 5: The approach prompts language models to generate a deductive reasoning form response, then extract the pathway to build a graph.