Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models

Fangzhi Xu, Zhiyong Wu, Qiushi Sun, Siyu Ren, Fei Yuan, Shuai Yuan, Qika Lin, Yu Qiao, Jun Liu

TL;DR

This work targets the NL-centric interface limitation of large language models by introducing Symbol-LLM, a symbol-centric foundation that learns across ~20 symbolic forms via 34 text-to-symbol tasks. A two-stage tuning framework—Injection Stage to learn symbolic knowledge and Infusion Stage to balance with general NL data—prevents forgetting and preserves NL capabilities, yielding Symbol-LLMBase and Symbol-LLMInstruct variants. Empirical results show substantial improvements in symbol-related tasks, competitive NL performance, and strong math-delegation abilities with symbol-driven reasoning, including OOD extrapolation. The paper also introduces Symbol-evol to diversify symbolic definitions and analyzes Alignment and Uniformity to understand the embedding structure, with open-source releases to foster further development of symbol-centric LLMs.

Abstract

Although Large Language Models (LLMs) demonstrate remarkable ability in processing and generating human-like text, they do have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language(e.g., chemical molecular formula). Injecting a collection of symbolic data directly into the training of LLMs can be problematic, as it disregards the synergies among different symbolic families and overlooks the need for a balanced mixture of natural and symbolic data. In this work, we tackle these challenges from both a data and framework perspective and introduce Symbol-LLM series models. First, we curated a data collection consisting of 34 tasks and incorporating approximately 20 distinct symbolic families, intending to capture the interrelations and foster synergies between symbols. Then, a two-stage tuning framework succeeds in injecting symbolic knowledge without loss of the generality ability. Extensive experiments on both symbol- and NL-centric tasks demonstrate the balanced and superior performances of Symbol-LLM series models. The project page is https://xufangzhi.github.io/symbol-llm-page/.

Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Models

TL;DR

This work targets the NL-centric interface limitation of large language models by introducing Symbol-LLM, a symbol-centric foundation that learns across ~20 symbolic forms via 34 text-to-symbol tasks. A two-stage tuning framework—Injection Stage to learn symbolic knowledge and Infusion Stage to balance with general NL data—prevents forgetting and preserves NL capabilities, yielding Symbol-LLMBase and Symbol-LLMInstruct variants. Empirical results show substantial improvements in symbol-related tasks, competitive NL performance, and strong math-delegation abilities with symbol-driven reasoning, including OOD extrapolation. The paper also introduces Symbol-evol to diversify symbolic definitions and analyzes Alignment and Uniformity to understand the embedding structure, with open-source releases to foster further development of symbol-centric LLMs.

Abstract

Although Large Language Models (LLMs) demonstrate remarkable ability in processing and generating human-like text, they do have limitations when it comes to comprehending and expressing world knowledge that extends beyond the boundaries of natural language(e.g., chemical molecular formula). Injecting a collection of symbolic data directly into the training of LLMs can be problematic, as it disregards the synergies among different symbolic families and overlooks the need for a balanced mixture of natural and symbolic data. In this work, we tackle these challenges from both a data and framework perspective and introduce Symbol-LLM series models. First, we curated a data collection consisting of 34 tasks and incorporating approximately 20 distinct symbolic families, intending to capture the interrelations and foster synergies between symbols. Then, a two-stage tuning framework succeeds in injecting symbolic knowledge without loss of the generality ability. Extensive experiments on both symbol- and NL-centric tasks demonstrate the balanced and superior performances of Symbol-LLM series models. The project page is https://xufangzhi.github.io/symbol-llm-page/.
Paper Structure (85 sections, 5 equations, 19 figures, 11 tables)

This paper contains 85 sections, 5 equations, 19 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Data Collection
  4. $\mathcal{D}_{s_1}$: the existing symbolic datasets and benchmarks
  5. $\mathcal{D}_{s_2}$: novel text-to-symbol pairs by prompting GPT-4
  6. $\mathcal{D}_{s_3}$: new samples generated by applying Symbol-evol strategy
  7. $\mathcal{D}_{g}$: general data
  8. Symbol-LLM
  9. Tuning Phase 1: Injection Stage
  10. Tuning Phase 2: Infusion Stage
  11. Testing Phase
  12. Experiments
  13. Symbolic Tasks
  14. Symbol-LLM largely enhances the symbol-related capabilities of LLM.
  15. The unified modeling helps Symbol-LLM successfully capture the intrinsic relationships between different symbols.
  16. ...and 70 more sections

Figures (19)

  • Figure 1: Overview of the data collection procedure. It involves three key sources: (1) existing benchmarks, (2) new data generated via prompting GPT-4, and (3) new data synthesized using the Symbol-evol strategy.
  • Figure 2: Overall pipeline of Symbol-LLM. (a) is two-stage tuning framework, Injection stage and Infusion stage. (b) is the test phase with comprehensive settings, symbolic tasks, general tasks, and downstream tasks under the Symbol+Delegation paradigm.
  • Figure 3: Visualization of Alignment-Uniformity. Both metrics are inversely related, which means a lower value indicates better performance.
  • Figure 4: LLaMA-2-Chat (13B)
  • Figure 5: Symbol-LLM (13B)
  • ...and 14 more figures