ChiEngMixBench: Evaluating Large Language Models on Spontaneous and Natural Chinese-English Code-Mixed Generation

TL;DR

ChiEngMixBench proposes the first scalable benchmark to evaluate Chinese–English code-mixing in authentic academic contexts, reframing code-mixing as a Cognitive Alignment problem assessed by Spontaneity and Naturalness. The authors construct a data pipeline from authentic communities to create Minimal Contrastive Pairs and an Expert Baseline, enabling an objective dual-dimensional evaluation of LLMs without over-reliance on translation metrics. They reveal a Terminology Layering Strategy aligned with Matrix Language Frame theory and demonstrate an Alignment Tax where instruction-tuned models suppress spontaneous term switching. Across 12 models, automated metrics and human validation show that authentic, domain-specific code-mixing differs markedly from translation-focused benchmarks and provide a robust resource for advancing cognitive-alignment research in multilingual LLMs. The work highlights practical implications for evaluating and guiding multilingual systems in expert domains, and opens source assets to support reproducibility and extension to other domains.

Abstract

Code-mixing is increasingly prevalent in interactions between humans and large language models, yet existing work often reduces it to a translation or convertibility problem, making it difficult to assess whether a model's switching behavior is context-appropriate and aligned with human conventions. We introduce ChiEngMixBench, the first benchmark designed to evaluate code-mixing ability in authentic community contexts, built upon a general construction pipeline that enables scalable dataset development across domains and bilingual pairs. ChiEngMixBench formulates code-mixing as a cognitive alignment problem, characterized by two complementary signals: Spontaneity and Naturalness. Empirical evaluation shows that our metrics can systematically distinguish code-mixing performance across models. Beyond benchmarking, we further uncover an implicitly emergent Terminology Layering Strategy, a phenomenon consistent with the Matrix Language Frame (MLF) theory, indicating structured cognitive alignment between multilingual large language models and human communication.

Figures (11)

• Figure 1: Contrast between Stochastic Substitution vs. Cognitive Alignment strategies.Panel A illustrates the pitfalls of random replacement (common in synthetic benchmarks), resulting in pragmatically awkward phrasing and broken syntax. Panel B demonstrates the focus of ChiEngMixBench: the spontaneous retention of high-entropy Anchor Terms (e.g., "API", "logits") adhering to the Principle of Least Effort and expert community norms.
• Figure 2: The ChiEngMixBench Construction Pipeline.Phase I: Data curation from authentic communities using MediaCrawler and strict cleaning. Phase II: Construction of the Spontaneity Subset (MCP) via hybrid filtering and the Naturalness Baseline via expert profiling. Phase III: Dual-dimensional evaluation of models.
• Figure 3: Overview of the Dual-Dimensional Evaluation Framework.Left (Internal): Spontaneity is measured via Logits Probing on Minimal Contrastive Pairs (MCP) to capture the "Relative Lift" against Contextual Inertia. Right (External): Naturalness is measured via Expert Deviation Penalty (EDP) on constrained generation to ensure semantic style and structural compliance.
• Figure 4: Naturalness Score & Penalty Breakdown. DeepSeek-V3 leads with minimal penalties. Instruct models suffer from severe Ratio Penalty (Red bar) due to monolingual collapse.
• Figure 5: Cognitive Alignment Map. Ideal models (Top-Right) balance Spontaneity and Naturalness. The "Valley of Collapse" highlights the Alignment Tax.