LCTG Bench: LLM Controlled Text Generation Benchmark

TL;DR

LCTG Bench introduces the first Japanese controllability benchmark for LLMs, addressing the lack of low-resource language coverage and a unified cross-task evaluation framework. It defines three generation tasks—Summarization, Ad Text Generation, and Pros & Cons—tested under four constraint perspectives: Format, Character Count, Keyword, and Prohibited Word, enabling robust cross-task model selection. The benchmark evaluates nine Japanese LLMs along with three multilingual models (notably GPT-4) and validates outputs with GPT-4-based quality checks, revealing a significant performance gap between GPT-4 and Japanese-specific models and highlighting challenges in character-count constraints. Practical implications include guidance for selecting models for real-world business use cases and a foundation for improving controllability and evaluation in future work.

Abstract

The rise of large language models (LLMs) has led to more diverse and higher-quality machine-generated text. However, their high expressive power makes it difficult to control outputs based on specific business instructions. In response, benchmarks focusing on the controllability of LLMs have been developed, but several issues remain: (1) They primarily cover major languages like English and Chinese, neglecting low-resource languages like Japanese; (2) Current benchmarks employ task-specific evaluation metrics, lacking a unified framework for selecting models based on controllability across different use cases. To address these challenges, this research introduces LCTG Bench, the first Japanese benchmark for evaluating the controllability of LLMs. LCTG Bench provides a unified framework for assessing control performance, enabling users to select the most suitable model for their use cases based on controllability. By evaluating nine diverse Japanese-specific and multilingual LLMs like GPT-4, we highlight the current state and challenges of controllability in Japanese LLMs and reveal the significant gap between multilingual models and Japanese-specific models.

Figures (11)

• Figure 1: Overview of LCTG Bench.
• Figure 2: An example of summarization prompt (Character count).
• Figure 3: An example of ad text generation prompt (Keyword).
• Figure 4: An example of pros & cons generation prompt (Prohibited word). In this example, "フリーランスとして働く (working as a freelancer)" is a pros & cons topic.
• Figure 5: An Example of LLM output in ad text generation that includes irrelevant explanations at the beginning and end of sentences: the explanations are included in the character count, making it impossible to measure the appropriate number of characters for task response.