Boolean-aware Attention for Dense Retrieval

TL;DR

BoolAttn extends attention with a Boolean aware mechanism that includes a Cue Predictor, a Scope Predictor, a Bias Predictor, and an Operator-gating mechanism to adjust attention via $S_{Boolean} = S_{and} + S_{or} - S_{not}$. It is integrated into BERT to form Bool-BERT and is pretrained on a GPT-generated dataset before evaluation on Quest and BoolQuestions. The approach yields improvements in Avg Recall@K and MRR@10, demonstrating that encoding explicit Boolean semantics enhances dense retrieval of Boolean queries. However, the method increases parameter count and relies on cue and scope predictions that can propagate errors, motivating future work on gating strategies and data-efficient pretraining.

Abstract

We present Boolean-aware attention, a novel attention mechanism that dynamically adjusts token focus based on Boolean operators (e.g., and, or, not). Our model employs specialized Boolean experts, each tailored to amplify or suppress attention for operator-specific contexts. A predefined gating mechanism activates the corresponding experts based on the detected Boolean type. Experiments on Boolean retrieval datasets demonstrate that integrating BoolAttn with BERT greatly enhances the model's capability to process Boolean queries.

Figures (3)

• Figure 1: Token-wise attention scores are adjusted based on Boolean type. Tokens affected by and will receive more attention while tokens affected by not will receive less attention score.
• Figure 2: Boolean-aware Attention architecture.
• Figure 3: Recall@100 across different query templates.