SOCIALITE-LLAMA: An Instruction-Tuned Model for Social Scientific Tasks

TL;DR

Socialite-Llama presents an instruction-tuned Llama2-7B model trained on 20 social-science classification tasks via a hand-crafted instruction corpus (SocialiteInstructions). The approach yields strong zero- and few-shot performance, outperforming the base Llama2 on all seen tasks and generalizing to 5 of 6 related tasks, while matching or surpassing a state-of-the-art multi-task fine-tuned DeBERTa on many tasks. The work demonstrates that instruction tuning can imbue LLMs with broader social understanding, highlighting substantial gains in affective, pragmatic, and normative dimensions. By releasing both the model and its instructions, the authors provide a resource for researchers to extend social-science NLP capabilities with reduced data and computation.

Abstract

Social science NLP tasks, such as emotion or humor detection, are required to capture the semantics along with the implicit pragmatics from text, often with limited amounts of training data. Instruction tuning has been shown to improve the many capabilities of large language models (LLMs) such as commonsense reasoning, reading comprehension, and computer programming. However, little is known about the effectiveness of instruction tuning on the social domain where implicit pragmatic cues are often needed to be captured. We explore the use of instruction tuning for social science NLP tasks and introduce Socialite-Llama -- an open-source, instruction-tuned Llama. On a suite of 20 social science tasks, Socialite-Llama improves upon the performance of Llama as well as matches or improves upon the performance of a state-of-the-art, multi-task finetuned model on a majority of them. Further, Socialite-Llama also leads to improvement on 5 out of 6 related social tasks as compared to Llama, suggesting instruction tuning can lead to generalized social understanding. All resources including our code, model and dataset can be found through bit.ly/socialitellama.