Empower Nested Boolean Logic via Self-Supervised Curriculum Learning

TL;DR

This work shows that state-of-the-art language models struggle with multi-nested boolean logic, a fundamental form of logical reasoning. It introduces Curriculum Logical Reasoning (Clr), a self-supervised training paradigm that gradually exposes models to nested boolean patterns, starting from simple patterns and progressing to harder ones. Using the BoolKill benchmark built on SciTail, Clr yields strong gains, enabling models to generalize to long-hops of logic and to serve as a beneficial initialization for broader logical tasks like ReClor and DREAM. The findings suggest boolean logic provides a solid foundation for robust logical reasoning and demonstrate curriculum-based training as an effective means to cultivate such capabilities in both mid-sized models and select large language models.

Abstract

Beyond the great cognitive powers showcased by language models, it is crucial to scrutinize whether their reasoning capabilities stem from strong generalization or merely exposure to relevant data. As opposed to constructing increasingly complex logic, this paper probes into the boolean logic, the root capability of a logical reasoner. We find that any pre-trained language models even including large language models only behave like a random selector in the face of multi-nested boolean logic, a task that humans can handle with ease. To empower language models with this fundamental capability, this paper proposes a new self-supervised learning method \textit{Curriculum Logical Reasoning} (\textsc{Clr}), where we augment the training data with nested boolean logic chain step-by-step, and program the training from simpler logical patterns gradually to harder ones. This new training paradigm allows language models to effectively generalize to much harder and longer-hop logic, which can hardly be learned through naive training. Furthermore, we show that boolean logic is a great foundation for improving the subsequent general logical tasks.

Figures (3)

• Figure 1: While language models are capable of handling a range of complex logical tasks, they do not perform well on more basic nested boolean logic.
• Figure 2: Overview of Curriculum Logical Reasoning.
• Figure 3: Boolean accuracy of different models with increasing numbers of nested boolean operations ($u_k$/$\tilde{u}_k$).