Automated Cloud Infrastructure-as-Code Reconciliation with AI Agents

TL;DR

The paper tackles infrastructure drift by introducing NSync, an agentic system that reconciles out-of-band changes back into Infrastructure-as-Code configurations. It leverages cloud API traces to infer intent, uses specialized tools to generate IaC patches, and employs continual learning via a per-project knowledge base to improve over time, all without performing unsafe live tests. A novel evaluation pipeline injects realistic drifts through AWS SSM runbooks and validates patches against ground truth, demonstrating substantial improvements in patch accuracy (up to 0.97 pass@3) and token efficiency over baselines, with further gains from continual learning. The work contributes a practical dataset of 372 drift scenarios across five Terraform projects, opens-source NSync, and outlines a generalizable framework for IaC reconciliation across clouds and IaC frameworks.

Abstract

Cloud infrastructure is managed through a mix of interfaces -- traditionally, cloud consoles, command-line interfaces (CLI), and SDKs are the tools of choice. Recently, Infrastructure-as-Code/IaC frameworks (e.g., Terraform) have quickly gained popularity. Unlike conventional tools, IaC~frameworks encode the infrastructure in a "source-of-truth" configuration. They are capable of automatically carrying out modifications to the cloud -- deploying, updating, or destroying resources -- to bring the actual infrastructure into alignment with the IaC configuration. However, when IaC is used alongside consoles, CLIs, or SDKs, it loses visibility into external changes, causing infrastructure drift, where the configuration becomes outdated, and later IaC operations may undo valid updates or trigger errors. We present NSync, an automated system for IaC reconciliation that propagates out-of-band changes back into the IaC program. Our key insight is that infrastructure changes eventually all occur via cloud API invocations -- the lowest layer for cloud management operations. NSync gleans insights from API traces to detect drift (i.e., non-IaC changes) and reconcile it (i.e., update the IaC configuration to capture the changes). It employs an agentic architecture that leverages LLMs to infer high-level intents from noisy API sequences, synthesize targeted IaC updates using specialized tools, and continually improve through a self-evolving knowledge base of past reconciliations. We further introduce a novel evaluation pipeline for injecting realistic drifts into cloud infrastructure and assessing reconciliation performance. Experiments across five real-world Terraform projects and 372 drift scenarios show that NSync outperforms the baseline both in terms of accuracy (from 0.71 to 0.97 pass@3) and token efficiency (1.47$\times$ improvement).

Figures (9)

• Figure 1: A concrete IaC example written in Terraform. (a) Code snippets of a Terraform configuration file; (b) Cloud infrastructure provisioned by the configuration files; (c) Cloud infrastructure after out-of-band change that adds two VMs; (d) Code patch generated by NSync
• Figure 2: Overview of NSync.
• Figure 3: Dataset generation pipeline. Mutated configurations are generated from a base configuration and natural-language scenario, validated through a deployment testing, and retained if successful.
• Figure 4: LLM annotation of a synthetic API trace with 190 mutating events.
• Figure 5: Distribution of tool usage across reconciliation progress. The x-axis denotes the normalized round position (tooluse_round / total_round) for each run. Figure \ref{['fig:tool_use_time']}a In NSync-NL, drift_report is used uniformly throughout the process, while self_critique is concentrated near the end. Figure \ref{['fig:tool_use_time']}b In NSync, similar patterns are observed, but knowledge tools exhibit distinct timing: knowledge_retrieval is typically invoked early to supply prior context, while knowledge_update appears near the end to record new insights.