Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Targeted Distillation for Sentiment Analysis

Yice Zhang, Guangyu Xie, Jingjie Lin, Jianzhu Bao, Qianlong Wang, Xi Zeng, Ruifeng Xu

TL;DR

The paper tackles the practicality gap in sentiment analysis by proposing a targeted distillation framework that decouples knowledge and alignment into KnowDist and ICLDist. KnowDist harvests sentiment knowledge via multi-perspective prompting, while ICLDist tunes the student’s ability to follow task instructions through diversified, few-shot prompts, enabling strong generalization to unseen tasks. Extensive experiments across multiple teacher–student pairs on the comprehensive SentiBench benchmark show substantial gains over generic distillation and reveal that smaller models can outperform larger originals, with broad task coverage. The work also offers a structured benchmark and thoughtful ablations to demonstrate the complementary roles of KnowDist and ICLDist and discusses practical considerations and limitations for future improvement.

Abstract

This paper explores targeted distillation methods for sentiment analysis, aiming to build compact and practical models that preserve strong and generalizable sentiment analysis capabilities. To this end, we conceptually decouple the distillation target into knowledge and alignment and accordingly propose a two-stage distillation framework. Moreover, we introduce SentiBench, a comprehensive and systematic sentiment analysis benchmark that covers a diverse set of tasks across 12 datasets. We evaluate a wide range of models on this benchmark. Experimental results show that our approach substantially enhances the performance of compact models across diverse sentiment analysis tasks, and the resulting models demonstrate strong generalization to unseen tasks, showcasing robust competitiveness against existing small-scale models.

Targeted Distillation for Sentiment Analysis

TL;DR

The paper tackles the practicality gap in sentiment analysis by proposing a targeted distillation framework that decouples knowledge and alignment into KnowDist and ICLDist. KnowDist harvests sentiment knowledge via multi-perspective prompting, while ICLDist tunes the student’s ability to follow task instructions through diversified, few-shot prompts, enabling strong generalization to unseen tasks. Extensive experiments across multiple teacher–student pairs on the comprehensive SentiBench benchmark show substantial gains over generic distillation and reveal that smaller models can outperform larger originals, with broad task coverage. The work also offers a structured benchmark and thoughtful ablations to demonstrate the complementary roles of KnowDist and ICLDist and discusses practical considerations and limitations for future improvement.

Abstract

This paper explores targeted distillation methods for sentiment analysis, aiming to build compact and practical models that preserve strong and generalizable sentiment analysis capabilities. To this end, we conceptually decouple the distillation target into knowledge and alignment and accordingly propose a two-stage distillation framework. Moreover, we introduce SentiBench, a comprehensive and systematic sentiment analysis benchmark that covers a diverse set of tasks across 12 datasets. We evaluate a wide range of models on this benchmark. Experimental results show that our approach substantially enhances the performance of compact models across diverse sentiment analysis tasks, and the resulting models demonstrate strong generalization to unseen tasks, showcasing robust competitiveness against existing small-scale models.

Paper Structure

This paper contains 25 sections, 2 equations, 5 figures, 24 tables.

Table of Contents

  1. Introduction
  2. Two-stage Distillation Framework
  3. In-Context Learning Distillation
  4. SentiBench
  5. Experiments
  6. Experimental Setup
  7. Main Results
  8. Analyis of Two-stage Optimization
  9. Ablation Studies
  10. Discussions
  11. Related Work
  12. Conclusions
  13. Distillation Prompts
  14. Prompts in Knowledge-Driven Distillation
  15. Prompts in In-Context Learning Distillation
  16. ...and 10 more sections

Figures (5)

  • Figure 1: The comparison of our distilled model with other small-scale models in terms of the average performance on SentiBench ($F_1$-score, %).
  • Figure 2: Illustration of our distillation process, consisting of four steps: data collection, prompt construction, corpus generation, and student model optimization.
  • Figure 3: Performance trend of the student model with varying volumes of distillation data (%). Here, performance refers to the average $F_1$-score on SentiBench.
  • Figure 4: Representative example for sentiment analysis.
  • Figure 5: Representative example for sentiment analysis.