Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient course recommendations with T5-based ranking and summarization

Thijmen Bijl, Niels van Weeren, Suzan Verberne

TL;DR

It is concluded that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval, and that quantization has a large effect on RankT5.

Abstract

In this paper, we implement and evaluate a two-stage retrieval pipeline for a course recommender system that ranks courses for skill-occupation pairs. The in-production recommender system BrightFit provides course recommendations from multiple sources. Some of the course descriptions are long and noisy, while retrieval and ranking in an online system have to be highly efficient. We developed a two-step retrieval pipeline with RankT5 finetuned on MSMARCO as re-ranker. We compare two summarizers for course descriptions: a LongT5 model that we finetuned for the task, and a generative LLM (Vicuna) with in-context learning. We experiment with quantization to reduce the size of the ranking model and increase inference speed. We evaluate our rankers on two newly labelled datasets, with an A/B test, and with a user questionnaire. On the two labelled datasets, our proposed two-stage ranking with automatic summarization achieves a substantial improvement over the in-production (BM25) ranker: nDCG@10 scores improve from 0.482 to 0.684 and from 0.447 to 0.844 on the two datasets. We also achieve a 40% speed-up by using a quantized version of RankT5. The improved quality of the ranking was confirmed by the questionnaire completed by 29 respondents, but not by the A/B test. In the A/B test, a higher clickthrough rate was observed for the BM25-ranking than for the proposed two-stage retrieval. We conclude that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval, and that quantization has a large effect on RankT5. In the online evaluation, however, other factors than relevance play a role (such as speed and interpretability of the retrieval results), as well as individual preferences.

Efficient course recommendations with T5-based ranking and summarization

TL;DR

It is concluded that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval, and that quantization has a large effect on RankT5.

Abstract

In this paper, we implement and evaluate a two-stage retrieval pipeline for a course recommender system that ranks courses for skill-occupation pairs. The in-production recommender system BrightFit provides course recommendations from multiple sources. Some of the course descriptions are long and noisy, while retrieval and ranking in an online system have to be highly efficient. We developed a two-step retrieval pipeline with RankT5 finetuned on MSMARCO as re-ranker. We compare two summarizers for course descriptions: a LongT5 model that we finetuned for the task, and a generative LLM (Vicuna) with in-context learning. We experiment with quantization to reduce the size of the ranking model and increase inference speed. We evaluate our rankers on two newly labelled datasets, with an A/B test, and with a user questionnaire. On the two labelled datasets, our proposed two-stage ranking with automatic summarization achieves a substantial improvement over the in-production (BM25) ranker: nDCG@10 scores improve from 0.482 to 0.684 and from 0.447 to 0.844 on the two datasets. We also achieve a 40% speed-up by using a quantized version of RankT5. The improved quality of the ranking was confirmed by the questionnaire completed by 29 respondents, but not by the A/B test. In the A/B test, a higher clickthrough rate was observed for the BM25-ranking than for the proposed two-stage retrieval. We conclude that T5-based re-ranking and summarization for online course recommendation can obtain much better effectiveness than single-step lexical retrieval, and that quantization has a large effect on RankT5. In the online evaluation, however, other factors than relevance play a role (such as speed and interpretability of the retrieval results), as well as individual preferences.
Paper Structure (37 sections, 5 figures, 10 tables)

This paper contains 37 sections, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Course recommendations
  4. Quantization of ranking models
  5. Data collection
  6. Course data sources
  7. Pre-processing
  8. Queries
  9. Evaluation data
  10. BrightFit IT dataset
  11. Query generation
  12. Document pool creation
  13. Relevance assessment
  14. BrightFit general skill dataset
  15. Methods
  16. ...and 22 more sections

Figures (5)

  • Figure 1: Overview of the tokenized input size distribution of the course titles combined with the descriptions of all BrightFit courses per provider. The horizontal line marks the input size of 512 tokens which is the maximum for T5.
  • Figure 2: Prompt used for Vicuna to summarize course descriptions
  • Figure 3: Funnel diagram for the events recorded during the BrightFit A/B test combined for the USA and Australia. The first value in each box indicates the total number of recorded events, the percentage indicates how many events of that type were recorded compared to the previous stage in the funnel.
  • Figure 4: An example of the two lists a user is shown for the query Customer Service for Front Desk Employee in the questionnaire.
  • Figure 5: Results of the conducted user questionnaire showing user preference for each of the 10 questions in the questionnaire. In total responses from 29 users were collected.