Multimodal Item Scoring for Natural Language Recommendation via Gaussian Process Regression with LLM Relevance Judgments

TL;DR

NLRec often relies on Dense Retrieval, which assumes a unimodal relevance peak near the NL query, but user satisfaction can arise from multiple relevant passages distributed across the embedding space. The authors present GPR-LLM, a Gaussian Process Regression framework that uses a small set of LLM relevance judgments to learn a multimodal relevance surface over passage embeddings, with an RBF kernel to capture multiple modes. A targeted $\epsilon$-greedy sampling strategy selects labeled passages, enabling GP-based passage scoring that generalizes to all passages and aggregates to item rankings. Across four NLRec benchmarks and two LLM backbones, GPR-LLM with the RBF kernel consistently surpasses DR, Cross-Encoder, and Pointwise LLM relevance scoring under the same labeling budget, achieving up to 65% improvements and demonstrating robustness to kernel choice and sampling strategy.

Abstract

Natural Language Recommendation (NLRec) generates item suggestions based on the relevance between user-issued NL requests and NL item description passages. Existing NLRec approaches often use Dense Retrieval (DR) to compute item relevance scores from aggregation of inner products between user request embeddings and relevant passage embeddings. However, DR views the request as the sole relevance label, thus leading to a unimodal scoring function centered on the query embedding that is often a weak proxy for query relevance. To better capture the potential multimodal distribution of the relevance scoring function that may arise from complex NLRec data, we propose GPR-LLM that uses Gaussian Process Regression (GPR) with LLM relevance judgments for a subset of candidate passages. Experiments on four NLRec datasets and two LLM backbones demonstrate that GPR-LLM with an RBF kernel, capable of modeling multimodal relevance scoring functions, consistently outperforms simpler unimodal kernels (dot product, cosine similarity), as well as baseline methods including DR, cross-encoder, and pointwise LLM-based relevance scoring by up to 65%. Overall, GPR-LLM provides an efficient and effective approach to NLRec within a minimal LLM labeling budget.

Figures (20)

• Figure 1: (Top) Dense retrieval (DR) methods assume a unimodal relevance scoring function concentrated near the user's NL request within the embedding space. (Bottom) In practice, the true relevance function is often multimodal, with relevant passages dispersed across distinct regions. The relevance scoring function for NL request "Family friendly cities for vacations" is multimodal as it covers multiple theme passages, including "Kid-friendly Attractions," "Outdoor and Nature Activities," and "Healthcare and Emergency Readiness."
• Figure 2: Overview of GPR-LLM. A small subset of $R \ll N$ passages is sampled from the full set of $N$ passages using an $\epsilon$-greedy sampling strategy and labeled using an LLM. A query-specific GPR model is then trained on this labeled subset to estimate relevance scores for all passages. Finally, item-level relevance scores are aggregated from the passage-level scores using \ref{['eq:item-level']}, and items are ranked accordingly to generate the recommendation list.
• Figure 3: RQ1: Kernel choices comparison with Greedy Sampling ($\epsilon = 0$) under varying LLM labeling budget.
• Figure 4: RQ2: Performance of GPR-LLM under $R=50$ with varying sampling sets. Each configuration labels an $\epsilon$ fraction of random passages from the top-$\tau$ non-DR set and a $(1-\epsilon)$ fraction from top DR-ranked passages.
• Figure 5: RQ4: Distribution of the top-100 passages from DR and GPR-LLM for example queries across all datasets. The embedding space is reduced to two dimensions using t-SNE for visualization. Passages ranked highest by DR (purple squares) cluster tightly around the query (red star), reflecting DR’s unimodal assumption. In contrast, GPR-LLM (blue circles) identifies passages dispersed across multiple regions, highlighting its ability to capture the multimodal relevance distribution hypothesized in \ref{['fig:multimodal-dense']}.