Multi-Layer Ranking with Large Language Models for News Source Recommendation

TL;DR

This paper tackles the problem of identifying credible information sources for news by analyzing quote-speaker patterns. It introduces NewsQuote, a dataset with 23,571 quote-speaker pairs, and presents an expert-retrieval pipeline that leverages both traditional candidate/document-based retrieval and a novel multi-layer LLM ranking with a filtering mechanism. Key contributions include the construction of NewsQuote, the implementation of probabilistic expert retrieval methods, and the development of a two-layer LLM-based ranking framework that improves predictive quality while mitigating popularity bias. The findings show that multi-layer LLM ranking enhances recall and ranking robustness, though it can trade off some precision; future work aims to broaden data sources and incorporate human evaluations for real-world deployment.

Abstract

To seek reliable information sources for news events, we introduce a novel task of expert recommendation, which aims to identify trustworthy sources based on their previously quoted statements. To achieve this, we built a novel dataset, called NewsQuote, consisting of 23,571 quote-speaker pairs sourced from a collection of news articles. We formulate the recommendation task as the retrieval of experts based on their likelihood of being associated with a given query. We also propose a multi-layer ranking framework employing Large Language Models to improve the recommendation performance. Our results show that employing an in-context learning based LLM ranker and a multi-layer ranking-based filter significantly improve both the predictive quality and behavioural quality of the recommender system.

Figures (1)

• Figure 1: Illustration of the LLM Ranker and the Multi-layer Ranking-based Filter