MARAGS: A Multi-Adapter System for Multi-Task Retrieval Augmented Generation Question Answering

TL;DR

This paper presents a multi-adapter retrieval augmented generation system (MARAGS) for Meta's Comprehensive RAG competition for KDD CUP 2024, which achieved 2nd place for Task 1 as well as 3rd place on Task 2.

Abstract

In this paper we present a multi-adapter retrieval augmented generation system (MARAGS) for Meta's Comprehensive RAG (CRAG) competition for KDD CUP 2024. CRAG is a question answering dataset contains 3 different subtasks aimed at realistic question and answering RAG related tasks, with a diverse set of question topics, question types, time dynamic answers, and questions featuring entities of varying popularity. Our system follows a standard setup for web based RAG, which uses processed web pages to provide context for an LLM to produce generations, while also querying API endpoints for additional information. MARAGS also utilizes multiple different adapters to solve the various requirements for these tasks with a standard cross-encoder model for ranking candidate passages relevant for answering the question. Our system achieved 2nd place for Task 1 as well as 3rd place on Task 2.

Figures (2)

• Figure 1: Pipeline for MARAGS. Each Llama in the figure represents a distinct LoRa model that can be rapidly swapped out during inference and is trained for its specific task. These tasks include the API call generation and the final question answering, for each task. Note, Task 1 does not include API documents in its final prompt.
• Figure 2: CRAG Score results on the test dataset calculated via manual assessment.