Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling

Blessed Guda, Gabrial Zencha Ashungafac, Lawrence Francis, Carlee Joe-Wong

TL;DR

This work tackles telecom MCQ QA with open-source, efficient LLMs by introducing QMOS, which combines a Question-Masked loss and an Option Batch-Shuffle strategy within an enhanced Retrieval-Augmented Generation framework. The approach leverages Phi-2 and Falcon-7B, incorporating multi-embedding retrieval, advanced prompt engineering, and parameter-efficient finetuning (LoRA/QLoRa) to boost reasoning over telecom standards documents. Results show Phi-2 achieving up to $84.65\%$ accuracy and Falcon-7B up to $49.93\%$ on TeleQnA, outperforming baselines and reducing reliance on proprietary models. The findings demonstrate that carefully engineered RAG pipelines and bias-mitigating training objectives can enable competitive telecom QA performance with significantly lower compute and data requirements, enabling practical deployment in constrained environments.

Abstract

Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.

QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling

TL;DR

This work tackles telecom MCQ QA with open-source, efficient LLMs by introducing QMOS, which combines a Question-Masked loss and an Option Batch-Shuffle strategy within an enhanced Retrieval-Augmented Generation framework. The approach leverages Phi-2 and Falcon-7B, incorporating multi-embedding retrieval, advanced prompt engineering, and parameter-efficient finetuning (LoRA/QLoRa) to boost reasoning over telecom standards documents. Results show Phi-2 achieving up to accuracy and Falcon-7B up to on TeleQnA, outperforming baselines and reducing reliance on proprietary models. The findings demonstrate that carefully engineered RAG pipelines and bias-mitigating training objectives can enable competitive telecom QA performance with significantly lower compute and data requirements, enabling practical deployment in constrained environments.

Abstract

Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.
Paper Structure (18 sections, 4 equations, 3 figures, 2 tables)

This paper contains 18 sections, 4 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Methodology
  4. Retrieval-Augmented Generation Architecture
  5. Splitting Documents into Chunks
  6. Creating chunk embeddings
  7. Chunk retrieval
  8. Model Prompt
  9. Falcon-7B
  10. QMOS Finetuning approach
  11. Phi-2 Finetuning
  12. QLoRa
  13. LoRa
  14. Option Batch-Shuffle Trick
  15. Evaluation
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Chunk retrieval process: The top k chunks are retrieved using the KNN algorithm. Scores for each chunk is obtained from the dot product $q\cdot K^T$. The retrieved chunks are then used to create the input prompt to the LLM
  • Figure 2: Loss and accuracy scores when fine-tuning the Phi-2 model with the standard next-token prediction objective
  • Figure 3: Loss and accuracy scores when fine-tuning the Phi-2 model and considering only the answers in the next-token prediction objective