Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion

Ziyao Xu, Houfeng Wang

TL;DR

This paper investigates whether large language models possess basic decomposition and composition capabilities for natural-to-formal language conversion ($N2F$). It introduces the DEDC framework, which semi-automatically constructs samples and tasks to decouple decomposition from composition and tests robustness to compositional gaps and counter-intuitive symbolic names. Base evaluations reveal pronounced decomposition deficiencies across models, with composition being relatively stronger, and show that compositional gaps and naming inversions further degrade performance. The framework enables systematic error analysis and provides a path toward targeted improvements in LLMs’ N2F capabilities, with potential for broad generalization to other N2F tasks.

Abstract

To achieve generalized and robust natural-to-formal language conversion (N2F), large language models (LLMs) need to have strong capabilities of decomposition and composition in N2F when faced with an unfamiliar formal language and be able to cope with compositional gaps and counter-intuitive symbolic names. To investigate whether LLMs have this set of basic capabilities in N2F, we propose the DEDC framework. This framework semi-automatically performs sample and task construction, allowing decoupled evaluation of the set of decomposition and composition capabilities of LLMs in N2F. Based on this framework, we evaluate and analyze the most advanced LLMs, and the main findings include that: (1) the LLMs are deficient in both decomposition and composition; (2) the LLMs show a wide coverage of error types that can be attributed to deficiencies in natural language understanding and the learning and use of symbolic systems; (3) compositional gaps and counter-intuitive symbolic names both affect the decomposition and composition of the LLMs. Our work provides a new perspective for investigating the basic capabilities of decomposition and composition of LLMs in N2F. The detailed analysis of deficiencies and attributions can help subsequent improvements of LLMs.

Investigating the (De)Composition Capabilities of Large Language Models in Natural-to-Formal Language Conversion

TL;DR

This paper investigates whether large language models possess basic decomposition and composition capabilities for natural-to-formal language conversion (). It introduces the DEDC framework, which semi-automatically constructs samples and tasks to decouple decomposition from composition and tests robustness to compositional gaps and counter-intuitive symbolic names. Base evaluations reveal pronounced decomposition deficiencies across models, with composition being relatively stronger, and show that compositional gaps and naming inversions further degrade performance. The framework enables systematic error analysis and provides a path toward targeted improvements in LLMs’ N2F capabilities, with potential for broad generalization to other N2F tasks.

Abstract

To achieve generalized and robust natural-to-formal language conversion (N2F), large language models (LLMs) need to have strong capabilities of decomposition and composition in N2F when faced with an unfamiliar formal language and be able to cope with compositional gaps and counter-intuitive symbolic names. To investigate whether LLMs have this set of basic capabilities in N2F, we propose the DEDC framework. This framework semi-automatically performs sample and task construction, allowing decoupled evaluation of the set of decomposition and composition capabilities of LLMs in N2F. Based on this framework, we evaluate and analyze the most advanced LLMs, and the main findings include that: (1) the LLMs are deficient in both decomposition and composition; (2) the LLMs show a wide coverage of error types that can be attributed to deficiencies in natural language understanding and the learning and use of symbolic systems; (3) compositional gaps and counter-intuitive symbolic names both affect the decomposition and composition of the LLMs. Our work provides a new perspective for investigating the basic capabilities of decomposition and composition of LLMs in N2F. The detailed analysis of deficiencies and attributions can help subsequent improvements of LLMs.
Paper Structure (28 sections, 4 figures, 8 tables)

This paper contains 28 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. DEDC Framework
  3. Sample Construction
  4. Task Construction
  5. Metric
  6. Decoupling
  7. Base Evaluation
  8. Models
  9. Results
  10. Error Analysis
  11. Evaluations with Additional Settings
  12. Compositional Gap
  13. Settings
  14. Results and Analysis
  15. Counter-intuitive Symbolic Name
  16. ...and 13 more sections

Figures (4)

  • Figure 1: An illustration of the sample and task construction of the DEDC framework. An example of the sample construction is shown on the left. On the right is an illustration of the task construction when the sample on the left is used as the test sample.
  • Figure 2: Six types of base graphs we identify for the sample construction of the STD framework.
  • Figure 3: An illustration of the compositional gap between the test sample and its demonstration samples. A and B indicate the different output types on the edges.
  • Figure 4: An illustration of the two types of settings for counter-intuitive symbolic names. The arrow indicates that a primitive with the meaning on the left end uses the name on the right end.