Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models

Changqun Li, Chaofan Ding, Kexin Luan, Xinhan Di

TL;DR

The document provides comprehensive, formal guidelines for preparing and submitting AAAI-compliant papers via PDFLaTeX, detailing required style adherence, file types, and submission procedures. It emphasizes using the 2025 aaai25 LaTeX style, embedding fonts, strict two-column US-letter formatting, and prohibited modifications or packages to ensure consistent production. It also prescribes the organization of front matter, abstract, sections, figures, tables, references, and ancillary materials, along with checks for copyright, acknowledgments, and ethical statements. Collectively, these guidelines enable reliable compilation, review integrity, and uniform appearance across AAAI proceedings, while clarifying processes for questions and resource references.

Abstract

Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low dimensional. Although LoRA has demonstrated commendable performance, there remains a significant performance gap between LoRA and full fine-tuning when learning new tasks. In this work, we propose Low-Rank Adaptation with Task-Relevant Feature Enhancement(LoRATRF) for enhancing task-relevant features from the perspective of editing neural network representations. To prioritize task-relevant features, a task-aware filter that selectively extracts valuable knowledge from hidden representations for the target or current task is designed. As the experiments on a vareity of datasets including NLU, commonsense reasoning and mathematical reasoning tasks demonstrates, our method reduces 33.71% parameters and achieves better performance on a variety of datasets in comparison with SOTA low-rank methods.

Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models

TL;DR

The document provides comprehensive, formal guidelines for preparing and submitting AAAI-compliant papers via PDFLaTeX, detailing required style adherence, file types, and submission procedures. It emphasizes using the 2025 aaai25 LaTeX style, embedding fonts, strict two-column US-letter formatting, and prohibited modifications or packages to ensure consistent production. It also prescribes the organization of front matter, abstract, sections, figures, tables, references, and ancillary materials, along with checks for copyright, acknowledgments, and ethical statements. Collectively, these guidelines enable reliable compilation, review integrity, and uniform appearance across AAAI proceedings, while clarifying processes for questions and resource references.

Abstract

Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. LoRA is one of the most widely used methods, which assumes that the optimization process is essentially low dimensional. Although LoRA has demonstrated commendable performance, there remains a significant performance gap between LoRA and full fine-tuning when learning new tasks. In this work, we propose Low-Rank Adaptation with Task-Relevant Feature Enhancement(LoRATRF) for enhancing task-relevant features from the perspective of editing neural network representations. To prioritize task-relevant features, a task-aware filter that selectively extracts valuable knowledge from hidden representations for the target or current task is designed. As the experiments on a vareity of datasets including NLU, commonsense reasoning and mathematical reasoning tasks demonstrates, our method reduces 33.71% parameters and achieves better performance on a variety of datasets in comparison with SOTA low-rank methods.

Paper Structure

This paper contains 63 sections, 3 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Copyright
  3. Formatting Requirements in Brief
  4. What Files to Submit
  5. Using LaTeX to Format Your Paper
  6. Document Preamble
  7. The Following Must Appear in Your Preamble
  8. Preparing Your Paper
  9. Commands and Packages That May Not Be Used
  10. Page Breaks
  11. Paper Size, Margins, and Column Width
  12. Column Width and Margins.
  13. Overlength Papers
  14. Type Font and Size
  15. Obtaining Type 1 Computer Modern for LaTeX.
  16. ...and 48 more sections

Figures (3)

  • Figure 1: Using the trim and clip commands produces fragile layers that can result in disasters (like this one from an actual paper) when the color space is corrected or the PDF combined with others for the final proceedings. Crop your figures properly in a graphics program -- not in LaTeX.
  • Figure 2: Adjusting the bounding box instead of actually removing the unwanted data resulted multiple layers in this paper. It also needlessly increased the PDF size. In this case, the size of the unwanted layer doubled the paper's size, and produced the following surprising results in final production. Crop your figures properly in a graphics program. Don't just alter the bounding box.
  • Figure 3: Example listing quicksort.hs