Beyond Training for Cultural Awareness: The Role of Dataset Linguistic Structure in Large Language Models

TL;DR

This work tackles cultural misalignment in LLMs by adopting a dataset-centric perspective and analyzing the linguistic properties of fine-tuning data. It introduces a language-specific PCA framework over ten lightweight metrics to derive three principal dataset dimensions (PC1–PC3) and then tests associations with cultural benchmarks across Arabic, Chinese, and Japanese using three model families. Through controlled subset interventions, the study finds that lexical and stylistic variation (PC3) yields the most robust, transferable improvements, while semantic density (PC1) and diversity (PC2) show model-dependent or neutral effects. The findings emphasize the importance of model-aware dataset design for multilingual cultural alignment and offer practical guidance for constructing culturally aware fine-tuning data. Overall, the work provides a nuanced view that dataset composition interacts with architecture in shaping cultural behavior, rather than providing universal data-quality metrics.

Abstract

The global deployment of large language models (LLMs) has raised concerns about cultural misalignment, yet the linguistic properties of fine-tuning datasets used for cultural adaptation remain poorly understood. We adopt a dataset-centric view of cultural alignment and ask which linguistic properties of fine-tuning data are associated with cultural performance, whether these properties are predictive prior to training, and how these effects vary across models. We compute lightweight linguistic, semantic, and structural metrics for Arabic, Chinese, and Japanese datasets and apply principal component analysis separately within each language. This design ensures that the resulting components capture variation among datasets written in the same language rather than differences between languages. The resulting components correspond to broadly interpretable axes related to semantic coherence, surface-level lexical and syntactic diversity, and lexical or structural richness, though their composition varies across languages. We fine-tune three major LLM families (LLaMA, Mistral, DeepSeek) and evaluate them on benchmarks of cultural knowledge, values, and norms. While PCA components correlate with downstream performance, these associations are strongly model-dependent. Through controlled subset interventions, we show that lexical-oriented components (PC3) are the most robust, yielding more consistent performance across models and benchmarks, whereas emphasizing semantic or diversity extremes (PC1-PC2) is often neutral or harmful.

Figures (16)

• Figure 1: Methodology
• Figure 2: Category-level contributions of diversity, lexical, semantic, and clustering metrics to principal components PC1–PC3 for Arabic, Japanese, and Chinese.
• Figure 3: Normalized PCA scores for all datasets in each language, projected onto the three main components (semantic relevance, diversity, lexical richness).
• Figure 4: Mean performance difference ( $\Delta$= subset - random) for High-PC and Low-PC subsets of equal size across PC1–PC3 and three model families, averaged over all base datasets and evaluation metrics. Zero indicates parity with random selection; positive values denote improvement and negative values degradation. PC1-PC2 rarely outperform random sampling, while PC3 shows more consistent, but model-dependent, gains.
• Figure : (a) AR–LLaMA