Distilling Fine-grained Sentiment Understanding from Large Language Models

TL;DR

This work tackles fine-grained sentiment analysis (FSA) by distilling LLM-driven sentiment understanding into compact SLMs to reduce inference costs while preserving interpretive capability. It introduces a two-prompt framework (analysis and rewriting) to extract rich sentiment content from LLMs and pretrain an SLM on the resulting corpus, complemented by a comprehensive FSA benchmark with hard samples and a robust evaluation protocol. Empirical results show distillation yields up to a 6.00% improvement in $F_1$ on FSA tasks, enables zero-shot SLM performance to match or surpass teacher models, and indicates data quantity can outweigh teacher quality in driving gains. The study provides practical insights for deploying FSA systems and offers valuable resources (code, data, weights) for the community.

Abstract

Fine-grained sentiment analysis (FSA) aims to extract and summarize user opinions from vast opinionated text. Recent studies demonstrate that large language models (LLMs) possess exceptional sentiment understanding capabilities. However, directly deploying LLMs for FSA applications incurs high inference costs. Therefore, this paper investigates the distillation of fine-grained sentiment understanding from LLMs into small language models (SLMs). We prompt LLMs to examine and interpret the sentiments of given reviews and then utilize the generated content to pretrain SLMs. Additionally, we develop a comprehensive FSA benchmark to evaluate both SLMs and LLMs. Extensive experiments on this benchmark reveal that: (1) distillation significantly enhances the performance of SLMs in FSA tasks, achieving a 6.00\% improvement in $F_1$-score, and the distilled model can outperform Llama-2-7b with only 220M parameters; (2) distillation equips SLMs with excellent zero-shot sentiment classification capabilities, enabling them to match or even exceed their teacher models. These results suggest that distillation from LLMs is a highly promising direction for FSA. We will release our code, data, and pretrained model weights at https://github.com/HITSZ-HLT/FSA-Distillation.

Figures (7)

• Figure 1: Knowledge distillation from LLMs enhances SLMs' capabilities in handling complex sentiment contexts.
• Figure 2: Illustration of our distillation process.
• Figure 3: The trend of LLMs' human evaluation scores with varying review lengths (scoring range 0-2 points).
• Figure 4: Scaling trends of review quantity and model size (average $F_1$-score on FSA datasets, %).
• Figure 5: Performance on data-scarce scenarios (average $F_1$-score on FSA datasets, %). Here, the teacher model is Mixtral-8x7b, and the number of reviews for distillation is 1 million.