Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey
Da Zheng, Lun Du, Junwei Su, Yuchen Tian, Yuqi Zhu, Jintian Zhang, Lanning Wei, Ningyu Zhang, Huajun Chen
TL;DR
The paper defines knowledge-augmented complex problem solving (CPS) for large language models from cognitive science and computational theory, identifying multi-step reasoning, domain knowledge, and result verification as core requirements. It surveys core methodologies, notably Chain-of-Thought and Tree-of-Thought frameworks, knowledge augmentation, verifier-based and symbolic/external verification, and agent-based architectures that integrate tools and human feedback. The discussion spans four domains—software engineering, mathematics, data science, and scientific research—highlighting domain-specific CPS challenges and the techniques developed to address them. It also outlines fundamental limitations such as data scarcity, verification difficulty, and high inference costs, and charts future directions including knowledge graphs, retrieval-augmented reasoning, and robust evaluation frameworks. Overall, the work provides a comprehensive blueprint for building robust CPS-enabled LLM systems with practical implications for real-world problem solving.
Abstract
Problem-solving has been a fundamental driver of human progress in numerous domains. With advancements in artificial intelligence, Large Language Models (LLMs) have emerged as powerful tools capable of tackling complex problems across diverse domains. Unlike traditional computational systems, LLMs combine raw computational power with an approximation of human reasoning, allowing them to generate solutions, make inferences, and even leverage external computational tools. However, applying LLMs to real-world problem-solving presents significant challenges, including multi-step reasoning, domain knowledge integration, and result verification. This survey explores the capabilities and limitations of LLMs in complex problem-solving, examining techniques including Chain-of-Thought (CoT) reasoning, knowledge augmentation, and various LLM-based and tool-based verification techniques. Additionally, we highlight domain-specific challenges in various domains, such as software engineering, mathematical reasoning and proving, data analysis and modeling, and scientific research. The paper further discusses the fundamental limitations of the current LLM solutions and the future directions of LLM-based complex problems solving from the perspective of multi-step reasoning, domain knowledge integration and result verification.