Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning

TL;DR

This work proposes a novel method called Semantic Knowledge Tuning (SK-Tuning) for prompt and prefix tuning that employs meaningful words instead of random tokens that offers a promising method for optimizing the efficiency and effectiveness of LLMs in processing language tasks.

Abstract

Large Language Models (LLMs) are gaining significant popularity in recent years for specialized tasks using prompts due to their low computational cost. Standard methods like prefix tuning utilize special, modifiable tokens that lack semantic meaning and require extensive training for best performance, often falling short. In this context, we propose a novel method called Semantic Knowledge Tuning (SK-Tuning) for prompt and prefix tuning that employs meaningful words instead of random tokens. This method involves using a fixed LLM to understand and process the semantic content of the prompt through zero-shot capabilities. Following this, it integrates the processed prompt with the input text to improve the model's performance on particular tasks. Our experimental results show that SK-Tuning exhibits faster training times, fewer parameters, and superior performance on tasks such as text classification and understanding compared to other tuning methods. This approach offers a promising method for optimizing the efficiency and effectiveness of LLMs in processing language tasks.

Figures (6)

• Figure 1: SK-Tuning approaches for Prefix (left) and Prompt (right). The dashed line represents the optimization path during the backward pass to the trainable adapter. Notably, in the context of prompt-tuning (on the right), the no sign signifies the discontinuation of the forward pass beyond a certain point. This is because we exclusively initialize layer-specific semantic information for the prompt, rendering the continuation of the forward pass unnecessary for the remaining layers.
• Figure 2: Comparison of memory efficiency (left) and training efficiency (right) across various PEFT methods. S-Prefix and S-Prompt represent SK-Tuning applied to prefix tuning and prompt tuning, respectively. The left chart shows the memory cost in GB, highlighting the model weights and optimizations, while the right chart displays the percentage of parameters, total training time in hours, and iteration time per second.
• Figure 3: Convergence Comparison for Token Classification
• Figure 4: Convergence Comparison for Sequence Classification
• Figure 5: Convergence Comparison for Sequence Classification