SteerEval: A Framework for Evaluating Steerability with Natural Language Profiles for Recommendation

TL;DR

SteerEval presents an end-to-end framework to evaluate steerability of natural-language profile-based recommender systems, addressing richer user-control actions beyond known attributes like genres. The framework combines profile creation (paragraph vs. sentence prompts), diverse steering interventions (template appends, LLM edits, full rewrites), and ranking methods (embedding similarity and LLM-based scoring) to measure how effectively user edits shift item rankings via a tag-based AUC metric $AUC_t$ and its change $\Delta AUC_t$. Experiments on MovieLens with TMDb metadata show that natural-language profile recommenders are generally steerable across many tags, with the strongest effects from rewrite-style interventions and stronger emphasis, though world-knowledge gaps and safety constraints reduce steerability for some trigger topics. The study offers design guidance, including augmenting metadata with steering-relevant information and preferring richer profile rewrites, while highlighting tradeoffs between steering strength and accuracy and suggesting directions for future work and broader domain applicability.

Abstract

Natural-language user profiles have recently attracted attention not only for improved interpretability, but also for their potential to make recommender systems more steerable. By enabling direct editing, natural-language profiles allow users to explicitly articulate preferences that may be difficult to infer from past behavior. However, it remains unclear whether current natural-language-based recommendation methods can follow such steering commands. While existing steerability evaluations have shown some success for well-recognized item attributes (e.g., movie genres), we argue that these benchmarks fail to capture the richer forms of user control that motivate steerable recommendations. To address this gap, we introduce SteerEval, an evaluation framework designed to measure more nuanced and diverse forms of steerability by using interventions that range from genres to content-warning for movies. We assess the steerability of a family of pretrained natural-language recommenders, examine the potential and limitations of steering on relatively niche topics, and compare how different profile and recommendation interventions impact steering effectiveness. Finally, we offer practical design suggestions informed by our findings and discuss future steps in steerable recommender design.

Figures (1)

• Figure 1: An illustration of the SteerEval steerability evaluation pipeline interwoven with the natural language recommendation process. The user profile generation process (A) creates an original user profile based on item metadata and user rating history (see section \ref{['sec:design_recommendation_creation']}). For a given original user profile and a desired steering action, the steering intervention (B) produces an updated steered profile (see section \ref{['sec:design_steering_intervention']}), and a ranking method (C) produces recommendations for each profile version (see section \ref{['sec:design_recommendation_ranking']}). Each pair of rankings is evaluated (D) for accuracy and steering effectiveness (see section \ref{['sec:design_steering_evaluation']}). Checkmarks denote the methods that we select for this example using our "default" pipeline.