Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Matters in LLM-Based Feature Extractor for Recommender? A Systematic Analysis of Prompts, Models, and Adaptation

Kainan Shi, Peilin Zhou, Ge Wang, Han Ding, Fei Wang

TL;DR

This paper tackles identifying which design choices in LLM-as-feature-extractor pipelines most drive performance in sequential recommender systems. It introduces RecXplore, a modular framework that decouples data processing, feature extraction, feature adaptation, and sequential modeling to enable controlled comparisons. Across four public datasets, the study shows simple, well-chosen designs—such as attribute flattening, mean pooling, CPT+SFT adaptation, and PCA+MoE feature adapters—yield substantial gains, up to 18.7% in NDCG@5 and 15.1% in HR@5 over strong baselines. The results support modular benchmarking as a practical path to robust, deployable LLM-enhanced recommendations and offer actionable guidelines for practitioners.

Abstract

Using Large Language Models (LLMs) to generate semantic features has been demonstrated as a powerful paradigm for enhancing Sequential Recommender Systems (SRS). This typically involves three stages: processing item text, extracting features with LLMs, and adapting them for downstream models. However, existing methods vary widely in prompting, architecture, and adaptation strategies, making it difficult to fairly compare design choices and identify what truly drives performance. In this work, we propose RecXplore, a modular analytical framework that decomposes the LLM-as-feature-extractor pipeline into four modules: data processing, semantic feature extraction, feature adaptation, and sequential modeling. Instead of proposing new techniques, RecXplore revisits and organizes established methods, enabling systematic exploration of each module in isolation. Experiments on four public datasets show that simply combining the best designs from existing techniques without exhaustive search yields up to 18.7% relative improvement in NDCG@5 and 12.7% in HR@5 over strong baselines. These results underscore the utility of modular benchmarking for identifying effective design patterns and promoting standardized research in LLM-enhanced recommendation.

What Matters in LLM-Based Feature Extractor for Recommender? A Systematic Analysis of Prompts, Models, and Adaptation

TL;DR

This paper tackles identifying which design choices in LLM-as-feature-extractor pipelines most drive performance in sequential recommender systems. It introduces RecXplore, a modular framework that decouples data processing, feature extraction, feature adaptation, and sequential modeling to enable controlled comparisons. Across four public datasets, the study shows simple, well-chosen designs—such as attribute flattening, mean pooling, CPT+SFT adaptation, and PCA+MoE feature adapters—yield substantial gains, up to 18.7% in NDCG@5 and 15.1% in HR@5 over strong baselines. The results support modular benchmarking as a practical path to robust, deployable LLM-enhanced recommendations and offer actionable guidelines for practitioners.

Abstract

Using Large Language Models (LLMs) to generate semantic features has been demonstrated as a powerful paradigm for enhancing Sequential Recommender Systems (SRS). This typically involves three stages: processing item text, extracting features with LLMs, and adapting them for downstream models. However, existing methods vary widely in prompting, architecture, and adaptation strategies, making it difficult to fairly compare design choices and identify what truly drives performance. In this work, we propose RecXplore, a modular analytical framework that decomposes the LLM-as-feature-extractor pipeline into four modules: data processing, semantic feature extraction, feature adaptation, and sequential modeling. Instead of proposing new techniques, RecXplore revisits and organizes established methods, enabling systematic exploration of each module in isolation. Experiments on four public datasets show that simply combining the best designs from existing techniques without exhaustive search yields up to 18.7% relative improvement in NDCG@5 and 12.7% in HR@5 over strong baselines. These results underscore the utility of modular benchmarking for identifying effective design patterns and promoting standardized research in LLM-enhanced recommendation.

Paper Structure

This paper contains 35 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sequential Recommendation
  4. LLM for Recommendation
  5. Problem Formulation
  6. RecXplore Framework
  7. Data Processing Module
  8. Feature Extraction Module
  9. LLM Fine-tuning Strategy.
  10. Feature Aggregation Strategy.
  11. Feature Adaptation Module
  12. Adaptation Architectures.
  13. Fusion with ID Embeddings.
  14. Sequential Modeling Module.
  15. Training and Inference
  16. ...and 20 more sections

Figures (4)

  • Figure 1: Two representative paradigms of applying LLMs in recommender systems.
  • Figure 2: Overview of the proposed RecXplore framework, which comprises four modules: Data Processing, Feature Extraction, Feature Adaptation, and Sequential Modeling. The figure illustrates the data flow from item attributes to final recommendations. For each module, we systematically examine representative design choices and their impact on recommendation performance.
  • Figure 3: Instruction templates for Keyword Extraction, Summarization, and Expansion. The placeholder description denotes the flattened item attributes.
  • Figure 4: Performance comparison of different feature extractors on Beauty and Games datasets.