Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking

TL;DR

This work scrutinizes whether open-source, decoder-only LLMs pretrained on unstructured text can serve as truly zero-shot query likelihood models for document ranking. It shows that such models exhibit strong zero-shot ranking capabilities, while instruction-tuning can hinder performance unless a question-generation task is included in fine-tuning data. The authors propose a two-stage pipeline—a zero-shot retriever followed by an LLM-based QLM re-ranker with BM25 interpolation—that achieves state-of-the-art results in zero-shot and few-shot settings on BEIR subsets without requiring training. They also provide extensive comparisons with baselines and other LLM variants, and release their code and prompts to support reproducibility and practical use in IR systems using open-source LLMs.

Abstract

In the field of information retrieval, Query Likelihood Models (QLMs) rank documents based on the probability of generating the query given the content of a document. Recently, advanced large language models (LLMs) have emerged as effective QLMs, showcasing promising ranking capabilities. This paper focuses on investigating the genuine zero-shot ranking effectiveness of recent LLMs, which are solely pre-trained on unstructured text data without supervised instruction fine-tuning. Our findings reveal the robust zero-shot ranking ability of such LLMs, highlighting that additional instruction fine-tuning may hinder effectiveness unless a question generation task is present in the fine-tuning dataset. Furthermore, we introduce a novel state-of-the-art ranking system that integrates LLM-based QLMs with a hybrid zero-shot retriever, demonstrating exceptional effectiveness in both zero-shot and few-shot scenarios. We make our codebase publicly available at https://github.com/ielab/llm-qlm.

Figures (1)

• Figure 1: Impact of $\alpha$ on TRECC dataset.