Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization

Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, Priyanka Agrawal

TL;DR

This paper proposes to improve zero-shot cross-lingual transfer by composing expert modules trained separately on language or task data via element-wise arithmetic operations to leverage unlabeled data and English labeled data.

Abstract

Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules.

Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization

TL;DR

This paper proposes to improve zero-shot cross-lingual transfer by composing expert modules trained separately on language or task data via element-wise arithmetic operations to leverage unlabeled data and English labeled data.

Abstract

Parameter-efficient fine-tuning (PEFT) using labeled task data can significantly improve the performance of large language models (LLMs) on the downstream task. However, there are 7000 languages in the world and many of these languages lack labeled data for real-world language generation tasks. In this paper, we propose to improve zero-shot cross-lingual transfer by composing language or task specialized parameters. Our method composes language and task PEFT modules via element-wise arithmetic operations to leverage unlabeled data and English labeled data. We extend our approach to cases where labeled data from more languages is available and propose to arithmetically compose PEFT modules trained on languages related to the target. Empirical results on summarization demonstrate that our method is an effective strategy that obtains consistent gains using minimal training of PEFT modules.
Paper Structure (25 sections, 5 equations, 2 figures, 7 tables)

This paper contains 25 sections, 5 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Language and Task Arithmetic
  3. Task-in-One-Language
  4. Composing via Language and Task Addition:
  5. Composing via Language and Task Addition and Subtraction:
  6. Task-in-Many-Languages
  7. Composing via Task-only Addition:
  8. Composing via Language and Task Addition and Subtraction:
  9. Experimental Setup
  10. Tasks and Datasets
  11. Training and Implementation Details
  12. Baselines
  13. Results and Discussion
  14. Task-in-One-Language
  15. Task-in-Many-Languages
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Illustration of our language and task arithmetic approach for zero-shot cross-lingual transfer using LoRA parameters learned on top of PaLM 2. (a) We train a task adapter using the summarization objective in En and language adapters using Prefix-LM in En and Pt. At inference time, a summary is generated in Pt, shown with a dotted frame (Subsection \ref{['subsection:arithmeticlangtask']}). (b) We add the weights of task adapters trained for summarization in languages similar to the target. We use the resulting vector for zero-shot summarization in the target language (Subsection \ref{['subsection:arithmetictask']}).
  • Figure 2: Relative ROUGE-2 improvement of our language & task arithmetic over the baseline (task adapter only). Our approach yields consistent improvements for most source-target language pairs.