On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform Setting

TL;DR

This paper examines the viability of combining multiple SE sentiment analysis tools in a voting classifier to improve performance across domains. It conducts two cross-platform experiments using three diverse tools trained on different SE data sets and five SE sentiment datasets, finding that gains are modest within known domains but often do not extend to cross-domain settings. The study highlights the strong influence of pre-training data, dataset labeling, and domain characteristics on cross-platform performance, showing that in many cases the best single tool outperforms the ensemble. Practical guidance emerges: when the domain is known, ensembles can yield small gains; when the domain is unknown, selecting the best tool per domain is typically more effective, and more gold-standard datasets are needed to better understand subjectivity in SE sentiment labeling.

Abstract

A positive working climate is essential in modern software development. It enhances productivity since a satisfied developer tends to deliver better results. Sentiment analysis tools are a means to analyze and classify textual communication between developers according to the polarity of the statements. Most of these tools deliver promising results when used with test data from the domain they are developed for (e.g., GitHub). But the tools' outcomes lack reliability when used in a different domain (e.g., Stack Overflow). One possible way to mitigate this problem is to combine different tools trained in different domains. In this paper, we analyze a combination of three sentiment analysis tools in a voting classifier according to their reliability and performance. The tools are trained and evaluated using five already existing polarity data sets (e.g. from GitHub). The results indicate that this kind of combination of tools is a good choice in the within-platform setting. However, a majority vote does not necessarily lead to better results when applying in cross-platform domains. In most cases, the best individual tool in the ensemble is preferable. This is mainly due to the often large difference in performance of the individual tools, even on the same data set. However, this may also be due to the different annotated data sets.