Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Framework for Measuring the Quality of Infrastructure-as-Code Scripts

Pandu Ranga Reddy Konala, Vimal Kumar, David Bainbridge, Junaid Haseeb

TL;DR

The paper tackles the lack of standardized quality assessment for Infrastructure as Code (IaC) by proposing a nine-category IaC code quality framework aligned with ISO/IEC 25010:2011 and validating it on Ansible repositories. It maps 95–plus IaC-specific attributes to the framework, then computes per-repository quality scores by normalizing attributes per 100 lines of code and aggregating them with equal weights, yielding a Total Score in the range 0–9. Empirically, analysis of 11,279 Ansible Galaxy repositories shows overall improvements in metadata, structure, security, and error handling over time, with declines in sophistication, functionality, and automation, suggesting evolving community priorities toward simpler, more maintainable IaC practices. The framework provides a standardized, extensible foundation for cross-tool IaC quality assessment, benchmarking, and further research, enabling practitioners and researchers to quantify and improve IaC script quality at scale.

Abstract

Infrastructure as Code (IaC) has become integral to modern software development, enabling automated and consistent configuration of computing environments. The rapid proliferation of IaC scripts has highlighted the need for better code quality assessment methods. This paper proposes a new IaC code quality framework specifically showcased for Ansible repositories as a foundation. By analyzing a comprehensive dataset of repositories from Ansible Galaxy, we applied our framework to evaluate code quality across multiple attributes. The analysis of our code quality metrics applied to Ansible Galaxy repositories reveal trends over time indicating improvements in areas such as metadata and error handling, while highlighting declines in others such as sophistication and automation. The framework offers practitioners a systematic tool for assessing and enhancing IaC scripts, fostering standardization and facilitating continuous improvement. It also provides a standardized foundation for further work into IaC code quality.

A Framework for Measuring the Quality of Infrastructure-as-Code Scripts

TL;DR

The paper tackles the lack of standardized quality assessment for Infrastructure as Code (IaC) by proposing a nine-category IaC code quality framework aligned with ISO/IEC 25010:2011 and validating it on Ansible repositories. It maps 95–plus IaC-specific attributes to the framework, then computes per-repository quality scores by normalizing attributes per 100 lines of code and aggregating them with equal weights, yielding a Total Score in the range 0–9. Empirically, analysis of 11,279 Ansible Galaxy repositories shows overall improvements in metadata, structure, security, and error handling over time, with declines in sophistication, functionality, and automation, suggesting evolving community priorities toward simpler, more maintainable IaC practices. The framework provides a standardized, extensible foundation for cross-tool IaC quality assessment, benchmarking, and further research, enabling practitioners and researchers to quantify and improve IaC script quality at scale.

Abstract

Infrastructure as Code (IaC) has become integral to modern software development, enabling automated and consistent configuration of computing environments. The rapid proliferation of IaC scripts has highlighted the need for better code quality assessment methods. This paper proposes a new IaC code quality framework specifically showcased for Ansible repositories as a foundation. By analyzing a comprehensive dataset of repositories from Ansible Galaxy, we applied our framework to evaluate code quality across multiple attributes. The analysis of our code quality metrics applied to Ansible Galaxy repositories reveal trends over time indicating improvements in areas such as metadata and error handling, while highlighting declines in others such as sophistication and automation. The framework offers practitioners a systematic tool for assessing and enhancing IaC scripts, fostering standardization and facilitating continuous improvement. It also provides a standardized foundation for further work into IaC code quality.

Paper Structure

This paper contains 13 sections, 3 equations, 4 figures, 7 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. COMPARING EXISTING IaC TOOLS AND PRACTICES
  4. RELATED WORK
  5. ABOUT ISO/IEC 25010:2011 SOFTWARE QUALITY STANDARDS
  6. RESEARCH GAPS IN STATE-OF-THE-ART STUDIES
  7. METHODOLOGY FOR MAPPING IaC CODE ATTRIBUTES
  8. COMPUTATION OF IaC CODE ATTRIBUTES
  9. TEMPORAL DYNAMICS OF CODE QUALITY
  10. DISCUSSION
  11. LIMITATIONS AND RECOMMENDATIONS
  12. CONCLUSION
  13. DATA AVAILABILITY

Figures (4)

  • Figure 1: Aligning of Our Proposed IaC Code Quality Framework to ISO/IEC 25010:2011 Characteristics
  • Figure 2: 6-Month Averages of Total Score Across Time
  • Figure 3: 6-Month Average of Category Scores Across Time
  • Figure 4: Observable Trends