Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confidence-aware Fine-tuning of Sequential Recommendation Systems via Conformal Prediction

Chen Wang, Fangxin Wang, Ruocheng Guo, Yueqing Liang, Philip S. Yu

TL;DR

A novel fine-tuning framework that integrates Conformal Prediction-based losses with CE loss to optimize accuracy alongside confidence that better aligns with widely used top-$K$ metrics is proposed, demonstrating that CPFT improves precision metrics and confidence calibration.

Abstract

In Sequential Recommendation Systems (SRecsys), traditional training approaches that rely on Cross-Entropy (CE) loss often prioritize accuracy but fail to align well with user satisfaction metrics. CE loss focuses on maximizing the confidence of the ground truth item, which is challenging to achieve universally across all users and sessions. It also overlooks the practical acceptability of ranking the ground truth item within the top-$K$ positions, a common metric in SRecsys. To address this limitation, we propose \textbf{CPFT}, a novel fine-tuning framework that integrates Conformal Prediction (CP)-based losses with CE loss to optimize accuracy alongside confidence that better aligns with widely used top-$K$ metrics. CPFT embeds CP principles into the training loop using differentiable proxy losses and computationally efficient calibration strategies, enabling the generation of high-confidence prediction sets. These sets focus on items with high relevance while maintaining robust coverage guarantees. Extensive experiments on five real-world datasets and four distinct sequential models demonstrate that CPFT improves precision metrics and confidence calibration. Our results highlight the importance of confidence-aware fine-tuning in delivering accurate, trustworthy recommendations that enhance user satisfaction.

Confidence-aware Fine-tuning of Sequential Recommendation Systems via Conformal Prediction

TL;DR

A novel fine-tuning framework that integrates Conformal Prediction-based losses with CE loss to optimize accuracy alongside confidence that better aligns with widely used top- metrics is proposed, demonstrating that CPFT improves precision metrics and confidence calibration.

Abstract

In Sequential Recommendation Systems (SRecsys), traditional training approaches that rely on Cross-Entropy (CE) loss often prioritize accuracy but fail to align well with user satisfaction metrics. CE loss focuses on maximizing the confidence of the ground truth item, which is challenging to achieve universally across all users and sessions. It also overlooks the practical acceptability of ranking the ground truth item within the top- positions, a common metric in SRecsys. To address this limitation, we propose \textbf{CPFT}, a novel fine-tuning framework that integrates Conformal Prediction (CP)-based losses with CE loss to optimize accuracy alongside confidence that better aligns with widely used top- metrics. CPFT embeds CP principles into the training loop using differentiable proxy losses and computationally efficient calibration strategies, enabling the generation of high-confidence prediction sets. These sets focus on items with high relevance while maintaining robust coverage guarantees. Extensive experiments on five real-world datasets and four distinct sequential models demonstrate that CPFT improves precision metrics and confidence calibration. Our results highlight the importance of confidence-aware fine-tuning in delivering accurate, trustworthy recommendations that enhance user satisfaction.
Paper Structure (50 sections, 16 equations, 3 figures, 7 tables)

This paper contains 50 sections, 16 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sequential Recommendation
  4. Confidence in Recommendation Systems
  5. Conformal Prediction
  6. Preliminaries
  7. Sequential Recommendation Systems
  8. Cross-Entropy (CE) Loss
  9. Conformal Prediction
  10. Split Conformal Prediction.
  11. Coverage Guarantee.
  12. Methodology
  13. Motivation and High-Level Idea
  14. Data Splitting
  15. Training Sequence.
  16. ...and 35 more sections

Figures (3)

  • Figure 1: Conceptual overview of CPFT. By combining confidence (via CP-based losses) with standard cross-entropy objectives, CPFT produces a smaller, high-confidence prediction set. The green shading reflects item-level confidence scores, which become more concentrated on relevant items (circles) under CP-based training.
  • Figure 2: Visualization of the proposed framework. Stage One: Standard training/fine-tuning for a sequential recommendation model. Stage Two: Data splitting partitions each user's sequence into training and calibration parts. Standard supervised loss (e.g., cross-entropy) is computed on the training sequences, while our CP-based losses (CPS and CPD) are computed on the calibration sequences. The inset (right) illustrates how $\mathcal{L}_{\text{CPS}}$ seeks to reduce the prediction set size, whereas $\mathcal{L}_{\text{CPD}}$ forces the top-$K$ items within the set to be closer to the ground truth item.
  • Figure 3: Performance Metrics of the SASRec Model During Training and CPFT. The model parameters were set with a learning rate of 0.0005, $\alpha=0.7$, $\beta=10$, $\gamma=1$, and $\text{TopKClosest}=10$ in the Scientific dataset. Lower values of the CPS loss and CPD loss indicate higher confidence.