Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Cognitive Robotics Matters: Lessons from OntoAgent and LLM Deployment in HARMONIC for Safety-Critical Robot Teaming

Sanjay Oruganti, Sergei Nirenburg, Marjorie McShane, Jesse English, Michael Roberts, Christian Arndt, Ramviyas Parasuraman, Luis Sentis

Abstract

Deploying embodied AI agents in the physical world demands cognitive capabilities for long-horizon planning that execute reliably, deterministically, and transparently. We present HARMONIC, a cognitive-robotic architecture that pairs OntoAgent, a content-centric cognitive architecture providing metacognitive self-monitoring, domain-grounded diagnosis, and consequence-based action selection over ontologically structured knowledge, with a modular reactive tactical layer. HARMONIC's modular design enables a functional evaluation of whether LLMs can replicate OntoAgent's cognitive capabilities, evaluated within the same robotic system under identical conditions. Six LLMs spanning frontier and efficient tiers replace OntoAgent in a collaborative maintenance scenario under native and knowledge-equalized conditions. Results reveal that LLMs do not consistently assess their own knowledge state before acting, causing downstream failures in diagnostic reasoning and action selection. These deficits persist even with equivalent procedural knowledge, indicating the issues are architectural rather than knowledge-based. These findings support the design of physically embodied systems in which cognitive architectures retain primary authority for reasoning, owing to their deterministic and transparent characteristics.

Why Cognitive Robotics Matters: Lessons from OntoAgent and LLM Deployment in HARMONIC for Safety-Critical Robot Teaming

Abstract

Deploying embodied AI agents in the physical world demands cognitive capabilities for long-horizon planning that execute reliably, deterministically, and transparently. We present HARMONIC, a cognitive-robotic architecture that pairs OntoAgent, a content-centric cognitive architecture providing metacognitive self-monitoring, domain-grounded diagnosis, and consequence-based action selection over ontologically structured knowledge, with a modular reactive tactical layer. HARMONIC's modular design enables a functional evaluation of whether LLMs can replicate OntoAgent's cognitive capabilities, evaluated within the same robotic system under identical conditions. Six LLMs spanning frontier and efficient tiers replace OntoAgent in a collaborative maintenance scenario under native and knowledge-equalized conditions. Results reveal that LLMs do not consistently assess their own knowledge state before acting, causing downstream failures in diagnostic reasoning and action selection. These deficits persist even with equivalent procedural knowledge, indicating the issues are architectural rather than knowledge-based. These findings support the design of physically embodied systems in which cognitive architectures retain primary authority for reasoning, owing to their deterministic and transparent characteristics.

Paper Structure

This paper contains 9 sections, 3 figures.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND AND RELATED WORK
  3. The HARMONIC Framework
  4. Strategic Layer: OntoAgent
  5. Tactical Layer and Interface
  6. Experimental Design
  7. Task Scenario and Platforms
  8. OntoAgent Reasoning Trace
  9. Task Initiation

Figures (3)

  • Figure 2: HARMONIC architecture with interchangeable strategic layers: (a) OntoAgent for structured metacognition and planning; (b) LLMAgent for tool-based reasoning; (c) a shared tactical layer connected via the same perception/action interface, enabling controlled comparison.
  • Figure 3: Overview of the HARMONIC architecture. The translation layer encodes perception and decodes action commands into blackboard parameters, and the tactical controller reads them and engages skills that drive the effectors. *Architecturally supported; not evaluated in the current study.
  • Figure 4: (1)--(4) Panels from the DEKADE UI. (1) Communication transcript between Daniel (human) and LEIA. (2) Robot's task agenda. (3) Complete reasoning transcripts. (4) Sample Vision Meaning Representations (VMRs) of detected objects. (5) Simulated ship environment scenes displaying the UGV. (6) Tabletop robot performing tasks. (7) FPV camera view from the manipulator. Task snapshots: (i) Initial position. (ii) Searching stores for thermostat. (iii) Picking up thermostat. (iv) Returning thermostat to Daniel. (v) Dropping thermostat at location.