SentiCSE: A Sentiment-aware Contrastive Sentence Embedding Framework with Sentiment-guided Textual Similarity

TL;DR

SentiCSE is proposed, a novel Sentiment-aware Contrastive Sentence Embedding framework for constructing sentiment representations via combined word-level and sentence-level objectives, whose quality is guaranteed by Sentiment-guided Textual Similarity (SgTS), a novel metric for evaluating the quality of sentiment representations.

Abstract

Recently, sentiment-aware pre-trained language models (PLMs) demonstrate impressive results in downstream sentiment analysis tasks. However, they neglect to evaluate the quality of their constructed sentiment representations; they just focus on improving the fine-tuning performance, which overshadows the representation quality. We argue that without guaranteeing the representation quality, their downstream performance can be highly dependent on the supervision of the fine-tuning data rather than representation quality. This problem would make them difficult to foray into other sentiment-related domains, especially where labeled data is scarce. We first propose Sentiment-guided Textual Similarity (SgTS), a novel metric for evaluating the quality of sentiment representations, which is designed based on the degree of equivalence in sentiment polarity between two sentences. We then propose SentiCSE, a novel Sentiment-aware Contrastive Sentence Embedding framework for constructing sentiment representations via combined word-level and sentence-level objectives, whose quality is guaranteed by SgTS. Qualitative and quantitative comparison with the previous sentiment-aware PLMs shows the superiority of our work. Our code is available at: https://github.com/nayohan/SentiCSE

Figures (8)

• Figure 1: The difference between sentence embedding from a semantic perspective and a sentiment perspective, which shows the necessity of focusing on embedding methods from a sentiment perspective for sentiment analysis, instead of the traditional semantic perspective.
• Figure 2: Comparison of STS and our SgTS. STS measures similarity of two sentences based on contextual semantics while SgTS judges similarity based on their sentiment polarities.
• Figure 3: The overview of SentiCSE. In the sentence-level objective, we take two anchors, one for 'positive' polarity and the other for 'negative' polarity. We then encourage the sentiment representations to be more close to the corresponding sentences belonging to the same polarity, and to be far from the corresponding sentences associated with different polarities. In the word-level objective, our model tries to predict the masked words as in conventional MLM. But the error signal is dropped if the model matches the polarity of the predicted word with the masked word, even though the word prediction was not correct.
• Figure 4: Visualized embeddings of the sentences included in each dataset. Each model is pre-trained on its own pretraining data, specificed in Table \ref{['tab:pre-trained datasets']}. PCA is used for visualization.
• Figure 5: (a): Correlation between our SgTS and few-shot accuracy. (b): Correlation between STS and few-shot accuracy. We can see there is a strong correlation between our SgTS and the accuracy of sentiment analysis ($\rho=0.7$ with $p$-value $< 0.01$) while there does not seem to be a correlation between STS and accuracy.