Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Framework for Ranking Content Providers Using Prompt Engineering and Self-Attention Network

Gosuddin Kamaruddin Siddiqi, Deven Santhosh Shah, Radhika Bansal, Askar Kamalov

TL;DR

The paper tackles the challenge of ranking Content Providers for topic-aware content recommendation by integrating explicit user feedback with content-based signals. It advances from a weak supervised baseline to a ground-truth–driven framework built through Prompt Engineering and SME-guided judgments, supplemented by a Self-Attention neural network trained on a Listwise Ranking objective to address cold-start providers. LightGBM Pairwise ranking is used as a comparative baseline, while the Self-Attention approach delivers stronger listwise performance and scalability across topics, languages, and regions. Online A/B experiments demonstrate improvements in alignment with brand missions, user engagement, and content quality, though it notes limitations for Local Content and suggests geography-aware extensions for further gains.

Abstract

This paper addresses the problem of ranking Content Providers for Content Recommendation System. Content Providers are the sources of news and other types of content, such as lifestyle, travel, gardening. We propose a framework that leverages explicit user feedback, such as clicks and reactions, and content-based features, such as writing style and frequency of publishing, to rank Content Providers for a given topic. We also use language models to engineer prompts that help us create a ground truth dataset for the previous unsupervised ranking problem. Using this ground truth, we expand with a self-attention based network to train on Learning to Rank ListWise task. We evaluate our framework using online experiments and show that it can improve the quality, credibility, and diversity of the content recommended to users.

A Framework for Ranking Content Providers Using Prompt Engineering and Self-Attention Network

TL;DR

The paper tackles the challenge of ranking Content Providers for topic-aware content recommendation by integrating explicit user feedback with content-based signals. It advances from a weak supervised baseline to a ground-truth–driven framework built through Prompt Engineering and SME-guided judgments, supplemented by a Self-Attention neural network trained on a Listwise Ranking objective to address cold-start providers. LightGBM Pairwise ranking is used as a comparative baseline, while the Self-Attention approach delivers stronger listwise performance and scalability across topics, languages, and regions. Online A/B experiments demonstrate improvements in alignment with brand missions, user engagement, and content quality, though it notes limitations for Local Content and suggests geography-aware extensions for further gains.

Abstract

This paper addresses the problem of ranking Content Providers for Content Recommendation System. Content Providers are the sources of news and other types of content, such as lifestyle, travel, gardening. We propose a framework that leverages explicit user feedback, such as clicks and reactions, and content-based features, such as writing style and frequency of publishing, to rank Content Providers for a given topic. We also use language models to engineer prompts that help us create a ground truth dataset for the previous unsupervised ranking problem. Using this ground truth, we expand with a self-attention based network to train on Learning to Rank ListWise task. We evaluate our framework using online experiments and show that it can improve the quality, credibility, and diversity of the content recommended to users.
Paper Structure (15 sections, 1 equation, 5 figures, 4 tables)

This paper contains 15 sections, 1 equation, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Problem Structuring
  4. Formulating solution
  5. Weak Supervised Approach
  6. Shortcomings
  7. Prompt Engineering and Self-Attention Network
  8. LightGBM Model - Pairwise Ranking
  9. Self-Attention Neural Network - ListWise Ranking
  10. Experiment Setup
  11. Experiment Results
  12. Experiment 1 (E1)
  13. Experiment 2 (E2)
  14. Conclusion
  15. Appendix

Figures (5)

  • Figure 1: NDCG gains when training with only positive samples.
  • Figure 2: NDCG gains when training with including negative samples with relevance label as 0.
  • Figure 3: Self-Attention Neural Network Architecture with ListWise Ranking
  • Figure 4: Self-Attention Neural Network Architecture with ListWise Ranking
  • Figure 5: Side-by-Side candidate generation for a content provider - Good Housekeeping. Left half (Treatment) of image showcases better alignment to Good Housekeeping brand statement - Good Housekeeping is your destination for everything from recipes to product reviews to home decor inspiration; than the right half (Control).