Attentive Reasoning Queries: A Systematic Method for Optimizing Instruction-Following in Large Language Models

TL;DR

Attentive Reasoning Queries (ARQs) introduce a structured, domain-informed reasoning framework for large language models, implemented within the Parlant system to improve instruction-following in complex, multi-turn conversations. ARQs reinstate critical guidelines and support intermediate reasoning through a predefined JSON schema of targeted queries, enabling better control and debuggability while reducing hallucinations. Empirical evaluation across 87 test scenarios shows ARQs outperforming Chain-of-Thought (CoT) and direct-response baselines, with strong gains in handling guideline re-application and hallucination prevention. The approach also reveals potential computational efficiency benefits when ARQ design is tailored to task structure, suggesting practical impact for deploying reliable business-facing conversational agents.

Abstract

We present Attentive Reasoning Queries (ARQs), a novel structured reasoning approach that significantly improves instruction-following in Large Language Models through domain-specialized reasoning blueprints. While LLMs demonstrate remarkable capabilities across diverse tasks, they often fail to maintain adherence to complex, use-case-specific instructions during multi-turn conversations, presenting challenges for business-critical applications. ARQs address this limitation by guiding LLMs through systematic reasoning steps with targeted queries that reinstate critical instructions and facilitate intermediate reasoning throughout the completion process. In extensive testing within Parlant, our framework for reliable customer-facing agents in which ARQs were born out of necessity, they achieved a 90.2% success rate across 87 test scenarios, outperforming both Chain-of-Thought reasoning (86.1%) and direct response generation (81.5%). ARQs showed particular strength in addressing persistent failure modes like guideline re-application and hallucination prevention. Our analysis also revealed that ARQs can potentially be more computationally efficient than free-form reasoning when carefully designed. These findings demonstrate that structured reasoning approaches provide effective mechanisms for controlling how LLMs process information and make decisions in complex scenarios.

Figures (6)

• Figure 1: Output examples: (1) using ARQs, (2) using CoT, and (3) performed without reasoning, all leading to the same final output. We assume the agent is equipped with information about nearby restaurants and their menus.
• Figure 2: An illustration of query-attention hypothesis: an ARQ asking the LLM to parrot the guideline before evaluating its relevance
• Figure 3: A diagram of the Parlant Engine, including how its different modules interact. Modules execute from left to right, where each module feeds the next one with its outputs. The message generator receives comprehensive inputs from all other modules, before tailoring a final response.
• Figure 4: Performance comparison across reasoning methods. Comprehensive tests evaluate all three modules (message generator, tool caller, and guideline proposer) working in conjunction.
• Figure 5: Average output token usage by module and reasoning method.