Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Models

TL;DR

This work proposes symbol-to-language (S2L), a tuning-free method that enables large language models to solve symbol-related problems with information expressed in natural language, and evaluates the S2L method using both API-based and open-source models over eight symbol-related tasks.

Abstract

Symbols (or more broadly, non-natural language textual representations) such as numerical sequences, molecular formulas, and table delimiters widely exist, playing important roles in various tasks such as abstract reasoning, chemical property prediction, and table question answering. Despite the impressive natural language comprehension capabilities of large language models (LLMs), their reasoning abilities for symbols remain inadequate, which could attributed to the difference between symbol representations and general natural languages. We propose symbol-to-language (S2L), a tuning-free method that enables large language models to solve symbol-related problems with information expressed in natural language. Specifically, S2L first converts the symbols involved to language-based representations, which can be implemented by prompting LLMs or leveraging external tools, then these language-based representations are integrated into the original problem via direct substitution or concatenation, serving as useful input information for LLMs. We evaluate the S2L method using both API-based (GPT-4, ChatGPT) and open-source (OpenChat) models over eight symbol-related tasks, ranging from symbol-only abstract reasoning to sentiment analysis in social media. Experimental results show that S2L consistently leads to superior performance. For example, by employing S2L for GPT-4, there can be average significant improvements of +21.9% and +9.5% for subtasks in 1D-ARC and Dyck language, respectively. Codes and data are available at https://github.com/THUNLP-MT/symbol2language.

Figures (13)

• Figure 1: Top: This illustration presents a symbol-related problem (move 1 pixel forward) from the 1D-ARC benchmark, comparing the responses of large language models using both conventional symbol-based and our language-based representations with symbol-to-language (S2L) conversion. Bottom: The S2L conversion has broad applicability across various scenarios involving diverse types of symbols.
• Figure 2: Example of applying symbol-to-language for 1D abstract reasoning task. We convert every sequence to its textual representation by prompting LLMs or using simple rules implemented in code, and then we transform the symbolized problem to language-enhanced or language-only representations.
• Figure 3: Example of applying symbol-to-language for Dyck language task. We convert every symbol (e.g., "[") to its textual description (e.g., "open square bracket") by prompting LLMs, and then transform the symbolized problem into language-based representation for both the question and ground truth.
• Figure 4: Example of applying symbol-to-language for property prediction. We convert each SMILES to its textual representation by prompting LLMs or using a translator.
• Figure 5: Example of applying symbol-to-language for emotional reranking of emojis. We convert each emoji to its language-based representation by prompting LLMs or using the names from the Unicode dictionary.