Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking

Malin Astrid Larsson, Harald Fosen Grunnaleite, Vinay Setty

TL;DR

The paper tackles scalable automated fact-checking by jointly training open-weight LLMs to perform claim detection, evidence re-ranking, and stance detection via multi-task learning, addressing the cost and reproducibility limits of large proprietary models. It introduces a unified framework that freezes the backbone and uses QLoRA adapters with task-specific heads, and explores CLS, CLM, and instruction-tuned configurations while varying task weights and training order. Results show substantial relative gains over zero-shot and few-shot baselines (up to 44%, 54%, and 31% in Macro-F1 for the three tasks) and competitive performance with single-task fine-tuning, with the Qwen3-4B model achieving the best balance of capacity and efficiency. The work provides empirical guidelines for practitioners, demonstrates the viability of a single open LLM for end-to-end AFC, and emphasizes sustainability, transparency, and reproducibility by using public data and releasing configurations and code.

Abstract

Large language models (LLMs) are reshaping automated fact-checking (AFC) by enabling unified, end-to-end verification pipelines rather than isolated components. While large proprietary models achieve strong performance, their closed weights, complexity, and high costs limit sustainability. Fine-tuning smaller open weight models for individual AFC tasks can help but requires multiple specialized models resulting in high costs. We propose \textbf{multi-task learning (MTL)} as a more efficient alternative that fine-tunes a single model to perform claim detection, evidence ranking, and stance detection jointly. Using small decoder-only LLMs (e.g., Qwen3-4b), we explore three MTL strategies: classification heads, causal language modeling heads, and instruction-tuning, and evaluate them across model sizes, task orders, and standard non-LLM baselines. While multitask models do not universally surpass single-task baselines, they yield substantial improvements, achieving up to \textbf{44\%}, \textbf{54\%}, and \textbf{31\%} relative gains for claim detection, evidence re-ranking, and stance detection, respectively, over zero-/few-shot settings. Finally, we also provide practical, empirically grounded guidelines to help practitioners apply MTL with LLMs for automated fact-checking.

One LLM to Train Them All: Multi-Task Learning Framework for Fact-Checking

TL;DR

The paper tackles scalable automated fact-checking by jointly training open-weight LLMs to perform claim detection, evidence re-ranking, and stance detection via multi-task learning, addressing the cost and reproducibility limits of large proprietary models. It introduces a unified framework that freezes the backbone and uses QLoRA adapters with task-specific heads, and explores CLS, CLM, and instruction-tuned configurations while varying task weights and training order. Results show substantial relative gains over zero-shot and few-shot baselines (up to 44%, 54%, and 31% in Macro-F1 for the three tasks) and competitive performance with single-task fine-tuning, with the Qwen3-4B model achieving the best balance of capacity and efficiency. The work provides empirical guidelines for practitioners, demonstrates the viability of a single open LLM for end-to-end AFC, and emphasizes sustainability, transparency, and reproducibility by using public data and releasing configurations and code.

Abstract

Large language models (LLMs) are reshaping automated fact-checking (AFC) by enabling unified, end-to-end verification pipelines rather than isolated components. While large proprietary models achieve strong performance, their closed weights, complexity, and high costs limit sustainability. Fine-tuning smaller open weight models for individual AFC tasks can help but requires multiple specialized models resulting in high costs. We propose \textbf{multi-task learning (MTL)} as a more efficient alternative that fine-tunes a single model to perform claim detection, evidence ranking, and stance detection jointly. Using small decoder-only LLMs (e.g., Qwen3-4b), we explore three MTL strategies: classification heads, causal language modeling heads, and instruction-tuning, and evaluate them across model sizes, task orders, and standard non-LLM baselines. While multitask models do not universally surpass single-task baselines, they yield substantial improvements, achieving up to \textbf{44\%}, \textbf{54\%}, and \textbf{31\%} relative gains for claim detection, evidence re-ranking, and stance detection, respectively, over zero-/few-shot settings. Finally, we also provide practical, empirically grounded guidelines to help practitioners apply MTL with LLMs for automated fact-checking.
Paper Structure (27 sections, 8 equations, 4 figures, 5 tables)

This paper contains 27 sections, 8 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Task Classification Heads
  4. Causal LM Heads
  5. Multi-Task Learning (MTL) Training
  6. Instruction Tuning
  7. Experimental Setup
  8. Datasets and Metrics
  9. Claim detection.
  10. Evidence Re-ranking
  11. Stance Detection
  12. Metrics.
  13. Models and Hyperparameters
  14. Experimental Results
  15. Task-Specific Evaluation
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Overview of the multi-task QLoRA fact-checking architecture.
  • Figure 2: Prompt used for instruction tuning. For few-shot variations, the prompt is appended with examples from the training data with one example per label.
  • Figure 3: Model scaling curve for Qwen3 models of different sizes.
  • Figure 4: Data scaling curve for Qwen3-4b.