Conceptual In-Context Learning and Chain of Concepts: Solving Complex Conceptual Problems Using Large Language Models

TL;DR

The paper addresses the challenge of solving complex conceptual problems with LLMs by injecting domain-specific conceptual information (CI) through shallow customization methods (SCMs). It introduces two SCMs, Conceptual In-context Learning (C-ICL) and Chain of Concepts (CoC), and formalizes CI as a directed-acyclic graph to structure domain knowledge for problem solving. In a data model engineering setting, CoC and C-ICL substantially improve answer correctness over ICL and CoT, reducing hallucinations and parroting, with gains of $30.6\%$ and $29.88\%$ respectively over CoT. The work demonstrates emergent problem solving when CI is organized hierarchically and discusses implications for domain-specific CP solving and potential extensions to multimodal CI.

Abstract

Science and engineering problems fall in the category of complex conceptual problems that require specific conceptual information (CI) like math/logic -related know-how, process information, or engineering guidelines to solve them. Large Language Models (LLMs) are promising agents to solve such complex conceptual problems due to their implications in advancing engineering and science tasks like assisted problem-solving. But vanilla LLMs, trained on open-world data, lack the necessary CI. In this work, we specifically explore shallow customization methods (SCMs) of LLMs for solving complex conceptual problems. We propose two novel SCM algorithms for LLM, to augment LLMs with CI and enable LLMs to solve complex conceptual problems: Conceptual In-Context Learning (C-ICL) and Chain of Concepts (CoC). The problem tackled in this paper is generation of proprietary data models in the engineering/industry domain based on conceptual information in data modelling guidelines. We evaluate our algorithms on varied sizes of the OpenAI LLMs against four evaluation metrics related to syntactic and semantic correctness, time and cost incurred. The proposed algorithms perform better than currently popular LLM SCMs like In-context Learning (ICL) and Chain of Thoughts (CoT). It was observed that as compared to CoT, response correctness increased by 30.6% and 29.88% for the new SCMs C-ICL and CoC respectively. Qualitative analysis suggests that the proposed new SCMs activate emergent capabilities in LLMs, previously unobserved in the existing SCMs. They make problem-solving processes more transparent and reduce hallucinations and the tendency of model responses to copy examples from prompts (parroting).

Figures (4)

• Figure 1: The LLM response is customised through 5 different SCMs using GPT-3.5-turbo-16k. Figure markings - right-aligned box are model input, left-aligned are model response, SCM prompts are marked in yellow, CP in blue, and for model's response: correct are marked in green and incorrect in red.
• Figure 2: Proposed SCM CoC based formulation of CI as a DAG: 1. General DAG of CI 2. CI DAG Example for proprietary data model generation CP.
• Figure 3: Sample response using GPT-3.5-turbo-16k for PDM* generation using five SCMs. The left aligned boxes are LLM's response and right ones are input prompts. Colour markings mean: SCM - yellow, CP - blue, errors in the generated PDM - red and correct features - green, JSON generated - grey.
• Figure 4: For the experiment matrix (Table \ref{['tab1']}) the metrics are plotted (from left to right and top to bottom) - $\theta_{C_1}$, $\theta_{C_2}$, $\theta_{C_3}$, $\theta_{C_4}$, $\theta_{time}$, $\theta_{cost}$, $\theta_{P}$

Theorems & Definitions (3)

• Example 1
• Example 2
• Example 3