Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Automating Personalization: Prompt Optimization for Recommendation Reranking

Chen Wang, Mingdai Yang, Zhiwei Liu, Pan Li, Linsey Pang, Qingsong Wen, Philip Yu

TL;DR

This work introduces AGP, a framework that optimizes the user-profile generation prompt to improve LLM-based reranking in recommender systems. By replacing manual prompts with a learned $p_{ ext{gen}}$ and leveraging position-based feedback within batched training, AGP achieves better generalization across users and more interpretable ranking signals. Experiments across Amazon, Yelp, and Goodreads demonstrate improved ranking quality (e.g., higher NDCG@10) and validate the importance of summarization, batch size, sequence length, and position-based feedback. The approach offers a scalable path to personalized reranking with reduced reliance on hand-tuned prompts, with potential extensions via reinforcement learning.

Abstract

Modern recommender systems increasingly leverage large language models (LLMs) for reranking to improve personalization. However, existing approaches face two key limitations: (1) heavy reliance on manually crafted prompts that are difficult to scale, and (2) inadequate handling of unstructured item metadata that complicates preference inference. We present AGP (Auto-Guided Prompt Refinement), a novel framework that automatically optimizes user profile generation prompts for personalized reranking. AGP introduces two key innovations: (1) position-aware feedback mechanisms for precise ranking correction, and (2) batched training with aggregated feedback to enhance generalization.

Automating Personalization: Prompt Optimization for Recommendation Reranking

TL;DR

This work introduces AGP, a framework that optimizes the user-profile generation prompt to improve LLM-based reranking in recommender systems. By replacing manual prompts with a learned and leveraging position-based feedback within batched training, AGP achieves better generalization across users and more interpretable ranking signals. Experiments across Amazon, Yelp, and Goodreads demonstrate improved ranking quality (e.g., higher NDCG@10) and validate the importance of summarization, batch size, sequence length, and position-based feedback. The approach offers a scalable path to personalized reranking with reduced reliance on hand-tuned prompts, with potential extensions via reinforcement learning.

Abstract

Modern recommender systems increasingly leverage large language models (LLMs) for reranking to improve personalization. However, existing approaches face two key limitations: (1) heavy reliance on manually crafted prompts that are difficult to scale, and (2) inadequate handling of unstructured item metadata that complicates preference inference. We present AGP (Auto-Guided Prompt Refinement), a novel framework that automatically optimizes user profile generation prompts for personalized reranking. AGP introduces two key innovations: (1) position-aware feedback mechanisms for precise ranking correction, and (2) batched training with aggregated feedback to enhance generalization.

Paper Structure

This paper contains 14 sections, 7 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Our Contributions.
  3. Methodology
  4. Problem Setup and Notation
  5. Challenges.
  6. User Profile Generation with a Learned Prompt
  7. Position-Based Evaluation and Feedback
  8. Batch Formation and Optimization
  9. Experiments
  10. Experimental Settings
  11. Results and Discussion
  12. Ablation Study
  13. Training Efficiency
  14. Conclusion

Figures (3)

  • Figure 1: The pipeline of the AutoGuidePrompt (AGP) framework, illustrating the process from user interaction history to optimized reranked lists.
  • Figure 2: Ablation study on summarization (left) and batch size/sequence length impact (right) using GPT-4o on the AMZ dataset. Summarization reduces overfitting, while batch size 10 and sequence length 5 yield optimal ranking performance.
  • Figure 3: Ablation study on Position-Based Feedback.