Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Task Arithmetic for Zero-Shot Information Retrieval

Marco Braga, Pranav Kasela, Alessandro Raganato, Gabriella Pasi

TL;DR

The paper tackles the challenge of zero-shot information retrieval under domain shifts that degrade LLM performance. It introduces Task Arithmetic, which constructs a domain Task Vector $\tau_D$ from the difference between a domain-tuned model and a pre-trained model, and injects it into an IR-tuned model to form $\Theta' = \Theta_T + \alpha \tau_D$, enabling training-free adaptation. Empirically, the approach is validated across eight biomedical, scientific, and multilingual datasets using six model families, achieving up to $18\%$ improvements in $NDCG@10$ and $15\%$ in $P@10$ and revealing nontrivial alpha sensitivity that benefits from lightweight tuning. The work highlights the practical value of reusing public domain-tuned LLMs for zero-shot IR, offering a scalable and resource-efficient path to domain and language transfer with publicly available tools and data.

Abstract

Large Language Models (LLMs) have shown impressive zero-shot performance across a variety of Natural Language Processing tasks, including document re-ranking. However, their effectiveness degrades on unseen tasks and domains, largely due to shifts in vocabulary and word distributions. In this paper, we investigate Task Arithmetic, a technique that combines the weights of LLMs pre-trained on different tasks or domains via simple mathematical operations, such as addition or subtraction, to adapt retrieval models without requiring additional fine-tuning. Our method is able to synthesize diverse tasks and domain knowledge into a single model, enabling effective zero-shot adaptation in different retrieval contexts. Extensive experiments on publicly available scientific, biomedical, and multilingual datasets show that our method improves state-of-the-art re-ranking performance by up to 18% in NDCG@10 and 15% in P@10. In addition to these empirical gains, our analysis provides insights into the strengths and limitations of Task Arithmetic as a practical strategy for zero-shot learning and model adaptation. We make our code publicly available at https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR.

Investigating Task Arithmetic for Zero-Shot Information Retrieval

TL;DR

The paper tackles the challenge of zero-shot information retrieval under domain shifts that degrade LLM performance. It introduces Task Arithmetic, which constructs a domain Task Vector from the difference between a domain-tuned model and a pre-trained model, and injects it into an IR-tuned model to form , enabling training-free adaptation. Empirically, the approach is validated across eight biomedical, scientific, and multilingual datasets using six model families, achieving up to improvements in and in and revealing nontrivial alpha sensitivity that benefits from lightweight tuning. The work highlights the practical value of reusing public domain-tuned LLMs for zero-shot IR, offering a scalable and resource-efficient path to domain and language transfer with publicly available tools and data.

Abstract

Large Language Models (LLMs) have shown impressive zero-shot performance across a variety of Natural Language Processing tasks, including document re-ranking. However, their effectiveness degrades on unseen tasks and domains, largely due to shifts in vocabulary and word distributions. In this paper, we investigate Task Arithmetic, a technique that combines the weights of LLMs pre-trained on different tasks or domains via simple mathematical operations, such as addition or subtraction, to adapt retrieval models without requiring additional fine-tuning. Our method is able to synthesize diverse tasks and domain knowledge into a single model, enabling effective zero-shot adaptation in different retrieval contexts. Extensive experiments on publicly available scientific, biomedical, and multilingual datasets show that our method improves state-of-the-art re-ranking performance by up to 18% in NDCG@10 and 15% in P@10. In addition to these empirical gains, our analysis provides insights into the strengths and limitations of Task Arithmetic as a practical strategy for zero-shot learning and model adaptation. We make our code publicly available at https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR.
Paper Structure (10 sections, 2 equations, 1 figure, 3 tables)

This paper contains 10 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Related Work and Motivation
  4. Task Arithmetic for Zero-Shot IR
  5. Experimental Setup
  6. Datasets and Evaluation Metrics
  7. Models and Baselines
  8. Results and Discussion
  9. Ablation Study
  10. Conclusion

Figures (1)

  • Figure 1: Proposed approach: Given a pre-trained LLM $\Theta_0$ and its domain-finetuned version $\Theta_D$, we compute the Task Vector $\tau_D$ as their parameter difference. To build a domain-specific IR model $\Theta'$, we add $\tau_D$ to an IR-finetuned model $\Theta_T$.