Thoughts without Thinking: Reconsidering the Explanatory Value of Chain-of-Thought Reasoning in LLMs through Agentic Pipelines

TL;DR

The paper investigates the explanatory value of Chain-of-Thought reasoning within agentic multi-LLM pipelines by implementing a perceptive task guidance system and evaluating it with Task and Org-Soc benchmarks. It adopts an offline evaluation framework using expert reviewers and a LLM-as-a-Judge to score outputs across multiple models and analyzes the content of CoT traces qualitatively. Key findings show that CoT reasoning does not reliably improve output quality or explainability; non-reasoning baselines perform better, and CoT traces can mislead via irrelevant content and the Einstellung Paradigm. The study highlights the need to rethink how reasoning traces are used for explanation in agentic AI and suggests integrating retrieval-based workflows to enhance explainability in practical, human-centered settings.

Abstract

Agentic pipelines present novel challenges and opportunities for human-centered explainability. The HCXAI community is still grappling with how best to make the inner workings of LLMs transparent in actionable ways. Agentic pipelines consist of multiple LLMs working in cooperation with minimal human control. In this research paper, we present early findings from an agentic pipeline implementation of a perceptive task guidance system. Through quantitative and qualitative analysis, we analyze how Chain-of-Thought (CoT) reasoning, a common vehicle for explainability in LLMs, operates within agentic pipelines. We demonstrate that CoT reasoning alone does not lead to better outputs, nor does it offer explainability, as it tends to produce explanations without explainability, in that they do not improve the ability of end users to better understand systems or achieve their goals.

Figures (4)

• Figure 1: Shows sample agentic flow depending on the type of input query from the user. Once the question is ingested, the lead planner generates a plan which is a sequence of agentic calls. The output of each of the agent in the flow is shown here.
• Figure 2: Shows the reviewer scores for the answers as rated by humans and LLM-as-a-judge on Task and Org-Soc questions. We observe that the reviewer scores for the non-reasoning models are better than their reasoning (Deepseek-) counterparts. We can also observe that the thoughts reviewer scores are weakly correlated with the answer reviewer scores.
• Figure 3: The figure shows our agentic implementation of the perceptive task guidance system. The agents are categorized into perceptors, planners and action agents. The agents are autonomous and rely either on LLMs or alternative deep learning models to accomplish their task. The agentic pipeline consists of fixed flow and dynamic flows. The dynamic flows involves planner invoked components. The plan generated by the planner is converted into agent invocations (Routers).
• Figure 4: Shows more examples of dynamic flows in the agentic flow. The examples show the input question from the users in the Perceptor. The Lead planner creates the plan and creates the Route which consists of agent calls. RAG consists of spec documents for each toy which is further chunked. The RAG module further converts the input context into a query to the database. The retrieved document along with the inputs is passed to the following agents. The Response generation module consists of planners which decides if the context is sufficient to answer the question or not. The answer planner then invokes the expert question generator or the answer generator. The responses are always verified before publishing the responses to the users.