"Newspaper Eat" Means "Not Tasty": A Taxonomy and Benchmark for Coded Languages in Real-World Chinese Online Reviews

TL;DR

Coded language in real-world Chinese reviews poses a significant challenge for NLP models. The authors introduce CodedLang, a dataset of 7,744 Chinese Google Maps reviews with 900 span-level coded-language annotations and a seven-category taxonomy encompassing phonetic, orthographic, and cross-lingual encodings. They benchmark LLMs on coded-language detection, classification, and review rating prediction, and conduct a phonetic analysis to understand pronunciation-based strategies. Results reveal that strong models still struggle to detect and interpret coded language, with phonetic distortions degrading downstream rating accuracy, underscoring a critical real-world NLP challenge and the need for more robust methods.

Abstract

Coded language is an important part of human communication. It refers to cases where users intentionally encode meaning so that the surface text differs from the intended meaning and must be decoded to be understood. Current language models handle coded language poorly. Progress has been limited by the lack of real-world datasets and clear taxonomies. This paper introduces CodedLang, a dataset of 7,744 Chinese Google Maps reviews, including 900 reviews with span-level annotations of coded language. We developed a seven-class taxonomy that captures common encoding strategies, including phonetic, orthographic, and cross-lingual substitutions. We benchmarked language models on coded language detection, classification, and review rating prediction. Results show that even strong models can fail to identify or understand coded language. Because many coded expressions rely on pronunciation-based strategies, we further conducted a phonetic analysis of coded and decoded forms. Together, our results highlight coded language as an important and underexplored challenge for real-world NLP systems.

Figures (5)

• Figure 1: Distribution of coded language categories in the annotated dataset.
• Figure 2: Rating-wise mean squared error (MSE) for review rating prediction under different textual representations. Results were predicted by GPT-5-mini. Sample sizes for each rating level were shown below the x-axis, reflecting imbalance across ratings.
• Figure 3: The average error rate for substitutions, deletions, and insertions per class using Pinyin representations, ranked by the total number of errors per class.
• Figure 4: The average review length by rating, comparing coded and non-coded reviews.
• Figure 5: The average error rate for substitutions, deletions, and insertions per class using IPA representations, ranked by the total number of errors per class.