Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories

Islem Bouzenia, Michael Pradel

TL;DR

This work tackles the opacity of decision-making in LLM-based software engineering agents by introducing a trajectory-based empirical framework that analyzes thought-action-result sequences from RepairAgent, Auto-CodeRover, and OpenHands. By unifying logs into a common trajectory representation and applying statistical, pattern-mining, and open-coding analyses, the study reveals how iteration length, token usage, action sequences, and semantic coherence relate to task success. Key findings show that successful trajectories balance exploration, explanation, and validation, while failures involve repetitive cycles and misalignment between reasoning and actions; semantic alignment between thoughts and actions is a strong predictor of efficiency and reliability. The dataset and annotation framework released by the authors aim to guide future research toward more transparent, robust, and efficient autonomous software engineering agents, with practical guidance for prompting strategies, failure diagnosis, and anti-pattern detection.

Abstract

Large Language Model (LLM)-based agents are increasingly employed to automate complex software engineering tasks, such as program repair and issue resolution. These agents operate by autonomously generating natural language thoughts, invoking external tools, and iteratively refining their solutions. Despite their widespread adoption, the internal decision-making processes of these agents remain largely unexplored, limiting our understanding of their operational dynamics and failure modes. In this paper, we present a large-scale empirical study of the thought-action-result trajectories of three state-of-the-art LLM-based agents: RepairAgent, AutoCodeRover, and OpenHands. We unify their interaction logs into a common format, capturing 120 trajectories and 2,822 LLM interactions focused on program repair and issue resolution. Our study combines quantitative analyses of structural properties, action patterns, and token usage with qualitative assessments of reasoning coherence and feedback integration. We identify key trajectory characteristics, such as iteration counts and token consumption, recurring action sequences, and the semantic coherence of thoughts, actions, and their results. Our findings reveal behavioral motifs and anti-patterns that distinguish successful from failed executions, providing actionable insights for improving agent design, including prompting strategies, failure diagnosis, and anti-pattern detection. We release our dataset and annotation framework to support further research on transparent and robust autonomous software engineering agents.

Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories

TL;DR

This work tackles the opacity of decision-making in LLM-based software engineering agents by introducing a trajectory-based empirical framework that analyzes thought-action-result sequences from RepairAgent, Auto-CodeRover, and OpenHands. By unifying logs into a common trajectory representation and applying statistical, pattern-mining, and open-coding analyses, the study reveals how iteration length, token usage, action sequences, and semantic coherence relate to task success. Key findings show that successful trajectories balance exploration, explanation, and validation, while failures involve repetitive cycles and misalignment between reasoning and actions; semantic alignment between thoughts and actions is a strong predictor of efficiency and reliability. The dataset and annotation framework released by the authors aim to guide future research toward more transparent, robust, and efficient autonomous software engineering agents, with practical guidance for prompting strategies, failure diagnosis, and anti-pattern detection.

Abstract

Large Language Model (LLM)-based agents are increasingly employed to automate complex software engineering tasks, such as program repair and issue resolution. These agents operate by autonomously generating natural language thoughts, invoking external tools, and iteratively refining their solutions. Despite their widespread adoption, the internal decision-making processes of these agents remain largely unexplored, limiting our understanding of their operational dynamics and failure modes. In this paper, we present a large-scale empirical study of the thought-action-result trajectories of three state-of-the-art LLM-based agents: RepairAgent, AutoCodeRover, and OpenHands. We unify their interaction logs into a common format, capturing 120 trajectories and 2,822 LLM interactions focused on program repair and issue resolution. Our study combines quantitative analyses of structural properties, action patterns, and token usage with qualitative assessments of reasoning coherence and feedback integration. We identify key trajectory characteristics, such as iteration counts and token consumption, recurring action sequences, and the semantic coherence of thoughts, actions, and their results. Our findings reveal behavioral motifs and anti-patterns that distinguish successful from failed executions, providing actionable insights for improving agent design, including prompting strategies, failure diagnosis, and anti-pattern detection. We release our dataset and annotation framework to support further research on transparent and robust autonomous software engineering agents.

Paper Structure

This paper contains 33 sections, 2 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data Collection
  4. Trajectory Parsing and Representation
  5. Statistical Analysis
  6. Categorizing Actions
  7. Mining Sequential Action Patterns
  8. Labeling Semantic Relationships
  9. Thought $t_i$ $\rightarrow$ action $a_i$
  10. Thought $t_i$ $\rightarrow$ thought $t_{i+1}$
  11. Action $a_i$ $\rightarrow$ action $a_{i+1}$
  12. Result $r_i$ $\rightarrow$ thought $t_{i+1}$
  13. Result $r_i$ $\rightarrow$ action $a_{i+1}$
  14. Results
  15. RQ1: Trajectory-Level Properties
  16. ...and 18 more sections

Figures (7)

  • Figure 1: Overview of our methodology.
  • Figure 2: Comparative trajectory length between successful and unsuccessful trajectories. Note that the y-axis differs between the agents.
  • Figure 3: Input, Output, and Total tokens over successful trajectories and unsuccessful ones. The y-axis differs for each agent.
  • Figure 4: Normalized action distribution across task execution phases for each agent.
  • Figure 5: Top 11 sequences of action 4-grams (Repair-Agent).
  • ...and 2 more figures

Theorems & Definitions (4)

  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4