LLMEmbed: Rethinking Lightweight LLM's Genuine Function in Text Classification

TL;DR

This paper tackles the inefficiency of prompt-based text classification with large LLMs by proposing LLMEmbed, a transfer-learning framework that uses embeddings from multiple lightweight LLMs. By extracting $\boldsymbol{\phi}_i^{(m,d_m)} = f(x_i|m,d_m)$ from backbones such as $\{\textbf{llama2}, \textbf{roberta}, \textbf{bert}\}$ across depths and fusing them into $\boldsymbol{\psi}_i = v(\{\boldsymbol{\phi}_i^{(m,d_m)}\})$ before classification with $g(\cdot|\theta_g)$ (with the backbone frozen), LLMEmbed achieves strong accuracy with substantially lower overhead. The approach attains state-of-the-art results among lightweight backbones and competitive performance relative to GPT-3-based CARP, while dramatically reducing training time, energy use, and external-token costs (e.g., using only $4\%$ of model parameters, $1.8\%$ electricity consumption, and $1.5\%$ runtime). This makes local deployment practical and scalable, offering a flexible way to leverage semantic embeddings for discriminative tasks beyond text classification, with potential extensions to other NLP problems and explainability-enhanced futures.

Abstract

With the booming of Large Language Models (LLMs), prompt-learning has become a promising method mainly researched in various research areas. Recently, many attempts based on prompt-learning have been made to improve the performance of text classification. However, most of these methods are based on heuristic Chain-of-Thought (CoT), and tend to be more complex but less efficient. In this paper, we rethink the LLM-based text classification methodology, propose a simple and effective transfer learning strategy, namely LLMEmbed, to address this classical but challenging task. To illustrate, we first study how to properly extract and fuse the text embeddings via various lightweight LLMs at different network depths to improve their robustness and discrimination, then adapt such embeddings to train the classifier. We perform extensive experiments on publicly available datasets, and the results show that LLMEmbed achieves strong performance while enjoys low training overhead using lightweight LLM backbones compared to recent methods based on larger LLMs, i.e. GPT-3, and sophisticated prompt-based strategies. Our LLMEmbed achieves adequate accuracy on publicly available benchmarks without any fine-tuning while merely use 4% model parameters, 1.8% electricity consumption and 1.5% runtime compared to its counterparts. Code is available at: https://github.com/ChunLiu-cs/LLMEmbed-ACL2024.

Figures (3)

• Figure 1: The principle of our proposed LLMEmbed. The left part shows how recent prompt-based methods work to classify texts. It can be seen that such a multi-step reasoning process can merely be performed serially, thus leading to high inference overhead. For comparison, instead of using LLM's content generation ability, we use the latent semantic embeddings extracted by LLMs to realize a much more effective adaptation for downstream classification tasks.
• Figure 2: The demonstration of our LLEmbed method. The whole pipeline is a kind of typical transfer learning framework in which the parameters of backbone are pre-trained and frozen, and only the parameters of classifier head is trained during training. We investigate to fuse the semantic embeddings extracted from llama2, roberta and bert. Moreover, for llama2, we extract the embeddings at multiple network depths, and fuse them later via pooling operators to improve the embeddings' generalized ability.
• Figure 3: The losses of training the downstream model.