Toward Automated Test Generation for Dockerfiles Based on Analysis of Docker Image Layers

TL;DR

The paper tackles the problem of generating automated tests for Dockerfiles, noting that traditional coverage-based methods are ill-suited because Dockerfiles lack branching. It introduces a processing-results-based approach that analyzes the resulting Docker image layers and Dockerfile instructions to determine test targets and generate CST-formatted tests. The method uses a five-step workflow—build, enumerate effects, select targets with heuristic scoring, determine test viewpoints, and acquire expectations—to produce tests that verify both metadata and file-level effects. Experimental evaluation on ten CST-using projects shows the approach can reproduce over 80% of developer-created tests and achieve high coverage of image-layer targets, though it highlights limitations such as contents verification and integration-style checks. The findings suggest practical potential for automated Dockerfile testing while outlining directions to improve scoring rules, extend test types, and assess degradation-detection effectiveness.

Abstract

Docker has gained attention as a lightweight container-based virtualization platform. The process for building a Docker image is defined in a text file called a Dockerfile. A Dockerfile can be considered as a kind of source code that contains instructions on how to build a Docker image. Its behavior should be verified through testing, as is done for source code in a general programming language. For source code in languages such as Java, search-based test generation techniques have been proposed. However, existing automated test generation techniques cannot be applied to Dockerfiles. Since a Dockerfile does not contain branches, the coverage metric, typically used as an objective function in existing methods, becomes meaningless. In this study, we propose an automated test generation method for Dockerfiles based on processing results rather than processing steps. The proposed method determines which files should be tested and generates the corresponding tests based on an analysis of Dockerfile instructions and Docker image layers. The experimental results show that the proposed method can reproduce over 80% of the tests created by developers.

Figures (9)

• Figure 1: Build execution example
• Figure 2: Example of test using CST for Dockerfile shown in \ref{['fig:dfile-sample']}
• Figure 3: Concept of proposed automated test generation
• Figure 4: Flow of test generation by proposed method
• Figure 5: Coverage of generated tests across all 10 projects