Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised Cycle Detection in Agentic Applications

Felix George, Harshit Kumar, Divya Pathak, Kaustabha Ray, Mudit Verma, Pratibha Moogi

TL;DR

The paper addresses hidden, cost-inefficient cycles in agentic applications powered by LLMs by proposing an unsupervised, hybrid cycle-detection framework. It integrates DAG-based edge-frequency analysis (CDDAG), call-stack subsequence repetition (CDCS), and embedding-based semantic similarity (CDSA), plus a hybrid stage that confirms potential cycles. On a LangGraph stock-market dataset with 1575 trajectories, the hybrid method delivers a Cycle F1 of 0.72 and Non-Cycle F1 of 0.99, outperforming individual modalities and reducing the need for expensive semantic checks. This work advances observability for autonomous agents by enabling unsupervised, real-time detection of bad cycles and lays groundwork for broader anomaly detection in agentic systems.

Abstract

Agentic applications powered by Large Language Models exhibit non-deterministic behaviors that can form hidden execution cycles, silently consuming resources without triggering explicit errors. Traditional observability platforms fail to detect these costly inefficiencies. We present an unsupervised cycle detection framework that combines structural and semantic analysis. Our approach first applies computationally efficient temporal call stack analysis to identify explicit loops and then leverages semantic similarity analysis to uncover subtle cycles characterized by redundant content generation. Evaluated on 1575 trajectories from a LangGraph-based stock market application, our hybrid approach achieves an F1 score of 0.72 (precision: 0.62, recall: 0.86), significantly outperforming individual structural (F1: 0.08) and semantic methods (F1: 0.28). While these results are encouraging, there remains substantial scope for improvement, and future work is needed to refine the approach and address its current limitations.

Unsupervised Cycle Detection in Agentic Applications

TL;DR

The paper addresses hidden, cost-inefficient cycles in agentic applications powered by LLMs by proposing an unsupervised, hybrid cycle-detection framework. It integrates DAG-based edge-frequency analysis (CDDAG), call-stack subsequence repetition (CDCS), and embedding-based semantic similarity (CDSA), plus a hybrid stage that confirms potential cycles. On a LangGraph stock-market dataset with 1575 trajectories, the hybrid method delivers a Cycle F1 of 0.72 and Non-Cycle F1 of 0.99, outperforming individual modalities and reducing the need for expensive semantic checks. This work advances observability for autonomous agents by enabling unsupervised, real-time detection of bad cycles and lays groundwork for broader anomaly detection in agentic systems.

Abstract

Agentic applications powered by Large Language Models exhibit non-deterministic behaviors that can form hidden execution cycles, silently consuming resources without triggering explicit errors. Traditional observability platforms fail to detect these costly inefficiencies. We present an unsupervised cycle detection framework that combines structural and semantic analysis. Our approach first applies computationally efficient temporal call stack analysis to identify explicit loops and then leverages semantic similarity analysis to uncover subtle cycles characterized by redundant content generation. Evaluated on 1575 trajectories from a LangGraph-based stock market application, our hybrid approach achieves an F1 score of 0.72 (precision: 0.62, recall: 0.86), significantly outperforming individual structural (F1: 0.08) and semantic methods (F1: 0.28). While these results are encouraging, there remains substantial scope for improvement, and future work is needed to refine the approach and address its current limitations.

Paper Structure

This paper contains 11 sections, 8 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation and Approach
  3. Detailed Methodology
  4. Cycle Detection Using DAG representation (CDDAG):
  5. Cycle Detection Using Call Stack representation (CDCS):
  6. Cycle detection using semantic analysis (CDSA):
  7. Hybrid Approach:
  8. Experiments and Discussion
  9. Dataset and Ground Truth Creation
  10. Results and Discussion
  11. Conclusions and Future Work

Figures (5)

  • Figure 1: Agent execution trajectory represented as span tree where nodes represent spans and edges represent the parent-child relationship. Edge label represent the temporal order.
  • Figure 2: Call Stack representation of Figure \ref{['fig:agent_trajectory']}
  • Figure 3: DAG representation of Figure \ref{['fig:agent_trajectory']}
  • Figure 4: Ground truth creation strategy for trajectory dataset
  • Figure 5: F1-Micro, Precesion and recall for structural approaches CDDAG, CDCS, and semantic similarity approach CDSA