Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Whole is Better than the Sum: Using Aggregated Demonstrations in In-Context Learning for Sequential Recommendation

Lei Wang, Ee-Peng Lim

TL;DR

This work tackles sequential recommendation with large language models by applying in-context learning and uncovering how prompt design, task alignment, and demonstration choice influence performance. It introduces LLMSRec-Syn, an aggregated-demonstration approach that combines multiple training-user demonstrations into a single, compact prompt, mitigating the prompt-length bottleneck and enhancing guidance to the model. Through a systematic study on instruction wording, task consistency, demonstration selection, and demonstration count, the authors show LLMSRec-Syn often outperforms standard one-shot and zero-shot LLM baselines and can rival supervised methods on sparse data. The method demonstrates robust gains across three datasets and suggests a promising direction for scalable, data-efficient LLM-based sequential recommendation, with limitations related to prompt engineering and potential personalization loss.

Abstract

Large language models (LLMs) have shown excellent performance on various NLP tasks. To use LLMs as strong sequential recommenders, we explore the in-context learning approach to sequential recommendation. We investigate the effects of instruction format, task consistency, demonstration selection, and number of demonstrations. As increasing the number of demonstrations in ICL does not improve accuracy despite using a long prompt, we propose a novel method called LLMSRec-Syn that incorporates multiple demonstration users into one aggregated demonstration. Our experiments on three recommendation datasets show that LLMSRec-Syn outperforms state-of-the-art LLM-based sequential recommendation methods. In some cases, LLMSRec-Syn can perform on par with or even better than supervised learning methods. Our code is publicly available at https://github.com/demoleiwang/LLMSRec_Syn.

The Whole is Better than the Sum: Using Aggregated Demonstrations in In-Context Learning for Sequential Recommendation

TL;DR

This work tackles sequential recommendation with large language models by applying in-context learning and uncovering how prompt design, task alignment, and demonstration choice influence performance. It introduces LLMSRec-Syn, an aggregated-demonstration approach that combines multiple training-user demonstrations into a single, compact prompt, mitigating the prompt-length bottleneck and enhancing guidance to the model. Through a systematic study on instruction wording, task consistency, demonstration selection, and demonstration count, the authors show LLMSRec-Syn often outperforms standard one-shot and zero-shot LLM baselines and can rival supervised methods on sparse data. The method demonstrates robust gains across three datasets and suggests a promising direction for scalable, data-efficient LLM-based sequential recommendation, with limitations related to prompt engineering and potential personalization loss.

Abstract

Large language models (LLMs) have shown excellent performance on various NLP tasks. To use LLMs as strong sequential recommenders, we explore the in-context learning approach to sequential recommendation. We investigate the effects of instruction format, task consistency, demonstration selection, and number of demonstrations. As increasing the number of demonstrations in ICL does not improve accuracy despite using a long prompt, we propose a novel method called LLMSRec-Syn that incorporates multiple demonstration users into one aggregated demonstration. Our experiments on three recommendation datasets show that LLMSRec-Syn outperforms state-of-the-art LLM-based sequential recommendation methods. In some cases, LLMSRec-Syn can perform on par with or even better than supervised learning methods. Our code is publicly available at https://github.com/demoleiwang/LLMSRec_Syn.
Paper Structure (19 sections, 4 equations, 10 figures, 4 tables)

This paper contains 19 sections, 4 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. What Makes In-Context Learning Work for Sequential Recommendation
  4. Experiment Setup
  5. In-Context Learning for Sequential Recommendation
  6. Wording of Instructions
  7. Task Consistency
  8. Selection of Demonstrations
  9. Number of Demonstrations
  10. In-Context Learning with Aggregated Demonstrations
  11. Experiments and Results
  12. Methods for Comparison
  13. Main Results
  14. Analysis of Aggregated Demonstrations
  15. Conclusion
  16. ...and 4 more sections

Figures (10)

  • Figure 1: The illustrative comparison of (a) supervised sequential recommendation method and (b) in-context learning based sequential recommendation method.
  • Figure 2: The overall framework of zero-shot, few-shot, and aggregated one-shot LLM-based sequential recommender systems.
  • Figure 3: Instruction Format options: (A) Full, (B) w/o preference alignment, (C) w/o watched movie focus, (D) w/o rank result format
  • Figure 4: Impact of task consistency between demonstrations and test instances. CIR: Candidate Inclusion Ratio of Demonstration Templates: (T1) Next-Item option; (T2) Contrasting Item Pair option; (T3) Ranked Items option.
  • Figure 5: Demonstration selection: (1) random selection; (2) overlapping interacted items; (3) cosine similarity between the SBERT embeddings of interacted item sequences; (4) cosine similarity between the LLM (OpenAI embeddings) of interacted item sequences; (5) cosine similarity using CL embeddings of interacted item sequences.
  • ...and 5 more figures