Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Fine-tuning and RAG: A Hybrid Strategy for Dynamic LLM Recommendation Updates

Changping Meng, Hongyi Ling, Jianling Wang, Yifan Liu, Shuzhou Zhang, Dapeng Hong, Mingyan Gao, Onkar Dalal, Ed Chi, Lichan Hong, Haokai Lu, Ningren Han

TL;DR

Dynamic shifts in user interests break static fine-tuning for LLM-powered recommenders. The authors compare fine-tuning and Retrieval-Augmented Generation (RAG) and propose a hybrid strategy that combines periodic fine-tuning with frequent RAG-based updates. They validate the approach with offline analyses and live A/B experiments on a billion-user platform, showing statistically significant improvements in user satisfaction metrics. This work offers a practical framework for maintaining high-quality LLM-powered recommendations in production and points to adaptive update cadences as a fruitful avenue for future research.

Abstract

Large Language Models (LLMs) empower recommendation systems through their advanced reasoning and planning capabilities. However, the dynamic nature of user interests and content poses a significant challenge: While initial fine-tuning aligns LLMs with domain knowledge and user preferences, it fails to capture such real-time changes, necessitating robust update mechanisms. This paper investigates strategies for updating LLM-powered recommenders, focusing on the trade-offs between ongoing fine-tuning and Retrieval-Augmented Generation (RAG). Using an LLM-powered user interest exploration system as a case study, we perform a comparative analysis of these methods across dimensions like cost, agility, and knowledge incorporation. We propose a hybrid update strategy that leverages the long-term knowledge adaptation of periodic fine-tuning with the agility of low-cost RAG. We demonstrate through live A/B experiments on a billion-user platform that this hybrid approach yields statistically significant improvements in user satisfaction, offering a practical and cost-effective framework for maintaining high-quality LLM-powered recommender systems.

Balancing Fine-tuning and RAG: A Hybrid Strategy for Dynamic LLM Recommendation Updates

TL;DR

Dynamic shifts in user interests break static fine-tuning for LLM-powered recommenders. The authors compare fine-tuning and Retrieval-Augmented Generation (RAG) and propose a hybrid strategy that combines periodic fine-tuning with frequent RAG-based updates. They validate the approach with offline analyses and live A/B experiments on a billion-user platform, showing statistically significant improvements in user satisfaction metrics. This work offers a practical framework for maintaining high-quality LLM-powered recommendations in production and points to adaptive update cadences as a fruitful avenue for future research.

Abstract

Large Language Models (LLMs) empower recommendation systems through their advanced reasoning and planning capabilities. However, the dynamic nature of user interests and content poses a significant challenge: While initial fine-tuning aligns LLMs with domain knowledge and user preferences, it fails to capture such real-time changes, necessitating robust update mechanisms. This paper investigates strategies for updating LLM-powered recommenders, focusing on the trade-offs between ongoing fine-tuning and Retrieval-Augmented Generation (RAG). Using an LLM-powered user interest exploration system as a case study, we perform a comparative analysis of these methods across dimensions like cost, agility, and knowledge incorporation. We propose a hybrid update strategy that leverages the long-term knowledge adaptation of periodic fine-tuning with the agility of low-cost RAG. We demonstrate through live A/B experiments on a billion-user platform that this hybrid approach yields statistically significant improvements in user satisfaction, offering a practical and cost-effective framework for maintaining high-quality LLM-powered recommender systems.
Paper Structure (11 sections, 4 figures)

This paper contains 11 sections, 4 figures.

Table of Contents

  1. Introduction
  2. Method
  3. Preliminary
  4. Fine-tune
  5. Granularity
  6. Retrieval Similarity
  7. Data Retrieval
  8. Results and Evaluation
  9. Offline Evaluation
  10. Live Experiment
  11. Conclusion

Figures (4)

  • Figure 1: Prompt for Novel Interest Prediction. Black lines are the fine-tuning prompt. Added Blue lines are the RAG version with injected recent watch history. Label is only used for fine-tuning, but not RAG
  • Figure 2: Refresh with Retrieval-Augmented Generation. Fine-tuning is refreshed monthly, while RAG is refreshed multiple times within a week.
  • Figure 3: Offline Evaluation.
  • Figure 4: Live experiment results for user metrics. The x-axis represents the date; the y-axis represents the relative difference (in percentage) between the treatment and control groups.