CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies

TL;DR

CultureBank introduces a scalable, community-derived pipeline to build a large, structured cultural knowledge base from online platforms, pairing descriptors with grounded scenarios for realistic LLM evaluation. It provides a detailed taxonomy, a three-stage construction pipeline, and quality controls, yielding 12K TikTok descriptors and 11K Reddit descriptors across thousands of groups and topics. The authors show that fine-tuning language models on CultureBank improves performance on culture-related tasks and enables zero-shot transfer to other benchmarks, while grounded evaluation reveals persistent gaps. They also offer a forward-looking set of recommendations for data diversity, evaluation methods, and training paradigms to advance culturally aware language technologies.

Abstract

To enhance language models' cultural awareness, we design a generalizable pipeline to construct cultural knowledge bases from different online communities on a massive scale. With the pipeline, we construct CultureBank, a knowledge base built upon users' self-narratives with 12K cultural descriptors sourced from TikTok and 11K from Reddit. Unlike previous cultural knowledge resources, CultureBank contains diverse views on cultural descriptors to allow flexible interpretation of cultural knowledge, and contextualized cultural scenarios to help grounded evaluation. With CultureBank, we evaluate different LLMs' cultural awareness, and identify areas for improvement. We also fine-tune a language model on CultureBank: experiments show that it achieves better performances on two downstream cultural tasks in a zero-shot setting. Finally, we offer recommendations based on our findings for future culturally aware language technologies. The project page is https://culturebank.github.io . The code and model is at https://github.com/SALT-NLP/CultureBank . The released CultureBank dataset is at https://huggingface.co/datasets/SALT-NLP/CultureBank .

Figures (7)

• Figure 1: Overview. Our goal is culturally-aware language technologies. To do so, we develop a pipeline and construct CultureBank with structured cultural descriptors. Each descriptor comes with a grounded scenario, persona, and question to help evaluate LLMs. We fine-tune a model on CultureBank and improve its performance on two cultural tasks.
• Figure 2: CultureBank construction pipeline. Starting from comments on online communities, we will (1) select culture-related comments and extract mentioned cultural descriptors, then (2) cluster these descriptors and summarize the clusters, and finally (3) post-process them to get agreement value and remove bad contents. Each step is validated by human evaluation.
• Figure 3: Workflow of grounded evaluation. We present the grounded question to an LLM and get an answer. Given the answer, we perform automatic evaluation and human evaluation.
• Figure 4: Human evaluation on win rates between different LLMs (50 examples per pair) evaluated by humans on cultural-awareness in grounded consulting scenarios. The two annotators achieved a Kappa score of 0.87.
• Figure 5: Detailed workflow of how we generate the scenario, persona, and question grounded on each cultural descriptor. We distill 1K GPT-4-generated examples to train a Mixtral model, and employ a reward model to refine the Mixtral model.