Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoning

Simeng Han, Tianyu Liu, Chuhan Li, Xuyuan Xiong, Arman Cohan

TL;DR

HybridMind tackles the dichotomy between natural-language and symbolic reasoning in large language models by introducing a meta-selector that adaptively chooses among NL, SL, SymbolNL, and NLSymbol for each problem. The approach combines a carefully designed data-generation pipeline, symbolic-language choices (Python for math, FOL for logic), and either finetuning (STaR, LoRA) or prompting to train the meta-selector. Empirical results on the MATH and FOLIO datasets show that dynamic meta-selection yields measurable improvements over single-method baselines, with finetuning excelling on MATH and prompting performing strongly on FOLIO, and notable gains on harder problem subsets. The work provides a scalable framework for flexible, cross-domain reasoning and releases code and data to support future research with practical implications for improving LLM reasoning under diverse task demands.

Abstract

LLMs approach logical and mathematical reasoning through natural or symbolic languages. While natural language offers human-accessible flexibility but suffers from ambiguity, symbolic reasoning provides precise, machine-executable inferences at the cost of strict domain constraints. We introduce HYBRIDMIND, an adaptive strategy that selects the optimal reasoning approach for each reasoning problem. Through extensive experiments, we evaluate both prompting-based approaches with state-of-the-art LLMs and fine-tuned open-source models. We find that fine-tuning LLaMA-3.1-8B-Instruct as a meta-selector outperforms GPT-4o's natural language reasoning by 4.4\% on FOLIO and 1.3\% on MATH. More notably, using GPT-3.5-turbo as a prompted meta-selector yields a 10\% improvement on FOLIO's challenging subset compared to GPT-4o. We will release our code and data to support future research.

HYBRIDMIND: Meta Selection of Natural Language and Symbolic Language for Enhanced LLM Reasoning

TL;DR

HybridMind tackles the dichotomy between natural-language and symbolic reasoning in large language models by introducing a meta-selector that adaptively chooses among NL, SL, SymbolNL, and NLSymbol for each problem. The approach combines a carefully designed data-generation pipeline, symbolic-language choices (Python for math, FOL for logic), and either finetuning (STaR, LoRA) or prompting to train the meta-selector. Empirical results on the MATH and FOLIO datasets show that dynamic meta-selection yields measurable improvements over single-method baselines, with finetuning excelling on MATH and prompting performing strongly on FOLIO, and notable gains on harder problem subsets. The work provides a scalable framework for flexible, cross-domain reasoning and releases code and data to support future research with practical implications for improving LLM reasoning under diverse task demands.

Abstract

LLMs approach logical and mathematical reasoning through natural or symbolic languages. While natural language offers human-accessible flexibility but suffers from ambiguity, symbolic reasoning provides precise, machine-executable inferences at the cost of strict domain constraints. We introduce HYBRIDMIND, an adaptive strategy that selects the optimal reasoning approach for each reasoning problem. Through extensive experiments, we evaluate both prompting-based approaches with state-of-the-art LLMs and fine-tuned open-source models. We find that fine-tuning LLaMA-3.1-8B-Instruct as a meta-selector outperforms GPT-4o's natural language reasoning by 4.4\% on FOLIO and 1.3\% on MATH. More notably, using GPT-3.5-turbo as a prompted meta-selector yields a 10\% improvement on FOLIO's challenging subset compared to GPT-4o. We will release our code and data to support future research.
Paper Structure (49 sections, 1 figure, 14 tables)

This paper contains 49 sections, 1 figure, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning with natural language and symbolic language
  4. Meta selection with LLMs
  5. Method
  6. Approaches to reasoning
  7. NL.
  8. SL.
  9. SymbolNL.
  10. NLSymbol.
  11. Choice of symbolic language
  12. Training data generation
  13. Experiments
  14. Datasets
  15. Model choice
  16. ...and 34 more sections

Figures (1)

  • Figure 1: Illustration of HybridMind. HybridMind enables the model to analyze the problem and choose the most suitable approach among NL, SL, NLSymbol, and SymbolNL. We have different reasoning paths for logic and mathematical reasoning problems. In this example, HybridMind selects SL for a mathematical reasoning problem, which leads to the right solution (105).