Fine-Tuning vs. RAG for Multi-Hop Question Answering with Novel Knowledge

TL;DR

This work addresses how knowledge-injection strategies influence open-domain multi-hop QA, with a focus on temporally novel information beyond pretraining. It systematically compares unsupervised finetuning, supervised finetuning, and retrieval-augmented generation (RAG) using three open-source 7B LLMs on two benchmarks (QASC and 2024 Events) under a unified multiple-choice evaluation, formalizing objectives with $L_{CLM}$ and $L_{CE}$. Findings show that unsupervised finetuning yields only modest gains, while RAG yields substantial improvements, especially for novel knowledge; supervised finetuning achieves the highest overall accuracy when task-specific labels are available. The results highlight the practical value of retrieval-based knowledge access for compositional reasoning and up-to-date information, offering actionable guidance for deploying lightweight LLMs on challenging multi-hop tasks.

Abstract

Multi-hop question answering is widely used to evaluate the reasoning capabilities of large language models (LLMs), as it requires integrating multiple pieces of supporting knowledge to arrive at a correct answer. While prior work has explored different mechanisms for providing knowledge to LLMs, such as finetuning and retrieval-augmented generation (RAG), their relative effectiveness for multi-hop question answering remains insufficiently understood, particularly when the required knowledge is temporally novel. In this paper, we systematically compare parametric and non-parametric knowledge injection methods for open-domain multi-hop question answering. We evaluate unsupervised fine-tuning (continual pretraining), supervised fine-tuning, and retrieval-augmented generation across three 7B-parameter open-source LLMs. Experiments are conducted on two benchmarks: QASC, a standard multi-hop science question answering dataset, and a newly constructed dataset of over 10,000 multi-hop questions derived from Wikipedia events in 2024, designed to test knowledge beyond the models' pretraining cutoff. Our results show that unsupervised fine-tuning provides only limited gains over base models, suggesting that continual pretraining alone is insufficient for improving multi-hop reasoning accuracy. In contrast, retrieval-augmented generation yields substantial and consistent improvements, particularly when answering questions that rely on temporally novel information. Supervised fine-tuning achieves the highest overall accuracy across models and datasets. These findings highlight fundamental differences in how knowledge injection mechanisms support multi-hop question answering and underscore the importance of retrieval-based methods when external or compositional knowledge is required.

Figures (1)

• Figure 1: Comparative Framework for Multi-Hop Question Answering across Knowledge Injection Methods