Less is More: Benchmarking LLM Based Recommendation Agents

TL;DR

This study addresses whether longer LLM context improves recommendations by conducting a within-subject benchmark across four production LLMs on the REGEN dataset with context windows from $5$ to $50$ items. It finds that recommendation quality remains flat in the range $0.16$--$0.23$ despite increasing context, while token usage grows roughly $8.2\times$ and latency varies by model, enabling up to $88\%$ inference-cost savings with minimal context. The authors contribute a universal pattern across providers, practical deployment guidelines favoring a minimal sufficient context (approximately $5$--$10$ items), and latency considerations for real-time systems. These results challenge the 'more context is better' paradigm and support cost-effective, scalable LLM-based recommendation ecosystems, particularly in agent-based setups that need to reserve context for planning and tool use.

Abstract

Large Language Models (LLMs) are increasingly deployed for personalized product recommendations, with practitioners commonly assuming that longer user purchase histories lead to better predictions. We challenge this assumption through a systematic benchmark of four state of the art LLMs GPT-4o-mini, DeepSeek-V3, Qwen2.5-72B, and Gemini 2.5 Flash across context lengths ranging from 5 to 50 items using the REGEN dataset. Surprisingly, our experiments with 50 users in a within subject design reveal no significant quality improvement with increased context length. Quality scores remain flat across all conditions (0.17--0.23). Our findings have significant practical implications: practitioners can reduce inference costs by approximately 88\% by using context (5--10 items) instead of longer histories (50 items), without sacrificing recommendation quality. We also analyze latency patterns across providers and find model specific behaviors that inform deployment decisions. This work challenges the existing ``more context is better'' paradigm and provides actionable guidelines for cost effective LLM based recommendation systems.

Figures (4)

• Figure 1: Recommendation quality vs. context length across four LLMs (n=50 users). All models show flat quality curves with overlapping confidence intervals, indicating no significant improvement with longer context.
• Figure 2: Cost-benefit analysis showing token cost multiplier (red) vs. quality change percentage (teal). Despite 8$\times$ increase in token costs, quality improvement is negligible ($\sim$0%).
• Figure 3: Quality score heatmap across models and context lengths. The uniform coloring (all values 0.16--0.23) demonstrates that no model benefits from longer context.
• Figure 4: Comprehensive analysis of history length impact across multiple dimensions. (A) Token consumption scales linearly with history length across all models. (B) Inference latency shows model-specific patterns. (C--D) Recommendation quality metrics remain stable regardless of context length.