OrLog: Resolving Complex Queries with LLMs and Probabilistic Reasoning

TL;DR

OrLog tackles the challenge of solving complex information needs with logical constraints by separating predicate-level plausibility estimation from the logical engine. It uses an LLM as an oracle to score atomic predicates in a decoding-free pass, then applies ProbLog to compute the posterior probability that an entity satisfies the query, enabling constraint-aware reranking. Across QUEST, OrLog consistently improves top-rank precision over monolithic LLM reasoning, especially for disjunctive queries, while drastically reducing token usage. The results demonstrate that principled neuro-symbolic reasoning can outperform end-to-end approaches in reliability and efficiency, and point to future directions in grounding, probabilistic logic, and robustness.

Abstract

Resolving complex information needs that come with multiple constraints should consider enforcing the logical operators encoded in the query (i.e., conjunction, disjunction, negation) on the candidate answer set. Current retrieval systems either ignore these constraints in neural embeddings or approximate them in a generative reasoning process that can be inconsistent and unreliable. Although well-suited to structured reasoning, existing neuro-symbolic approaches remain confined to formal logic or mathematics problems as they often assume unambiguous queries and access to complete evidence, conditions rarely met in information retrieval. To bridge this gap, we introduce OrLog, a neuro-symbolic retrieval framework that decouples predicate-level plausibility estimation from logical reasoning: a large language model (LLM) provides plausibility scores for atomic predicates in one decoding-free forward pass, from which a probabilistic reasoning engine derives the posterior probability of query satisfaction. We evaluate OrLog across multiple backbone LLMs, varying levels of access to external knowledge, and a range of logical constraints, and compare it against base retrievers and LLM-as-reasoner methods. Provided with entity descriptions, OrLog can significantly boost top-rank precision compared to LLM reasoning with larger gains on disjunctive queries. OrLog is also more efficient, cutting mean tokens by $\sim$90\% per query-entity pair. These results demonstrate that generation-free predicate plausibility estimation combined with probabilistic reasoning enables constraint-aware retrieval that outperforms monolithic reasoning while using far fewer tokens.

Figures (2)

• Figure 1: Overview of OrLog, a neuro-symbolic framework for resolving complex entity-seeking queries. For a given query five steps are followd. Step 1: a retriever narrows the search space to candidate entities $(R_q)$. Step 2: a semantic parser decomposes the query into atomic predicates and a logical form. Step 3: predicate priors are elicited from an LLM via a truth-valuation prompt, using a single decoding-free forward pass. Step 4: these priors instantiate a probabilistic program in ProbLog, which computes posterior query satisfaction for each entity. Step 5: Candidates are reranked by their inferred probabilities.
• Figure 2: Mean $\Delta$P@1 between OrLog and LLM-as-reasoner baseline across query‐template structures in QUEST, in the informed knowledge access setting. Each bar shows the average per-query gain or loss for a specific template, with green indicating OrLog outperforming the LLM-as-reasoner baseline, and red indicating the reverse. Left and right panels correspond to different underlying LLMs: Llama-3.3-70B-Instruct and Qwen-2.5-7B-Instruct. Larger deviations highlight structural patterns where symbolic probabilistic reasoning offers a clear advantage relative to reasoning purely with LLMs.