The challenge of generating and evolving real-life like synthetic test data without accessing real-world raw data -- a Systematic Review

TL;DR

This systematic review investigates how to generate and evolve synthetic test data that resemble real-world government data without relying on actual raw data, a need driven by privacy regulations and the decentralized nature of e-Government systems. An extensive search across IEEE Xplore, ACM DL, and Scopus yielded 1013 results, with 37 primary studies analyzed to categorize methods for synthetic data generation and to assess the relatively rare area of data evolution. The findings show that most approaches still require some form of real data input or domain-specific artifacts, and very few efforts address evolving synthetic data over time; two notable studies from Oslo/Testify hint at evolution but do not clearly demonstrate preservation of essential data attributes. The work highlights a critical gap for Digital Government Solutions: robust, evolvable, fully synthetic data generation that can mirror real-life data without exposure, underscoring the need for future research and validated metrics in this domain.

Abstract

Background: High-level system testing of applications that use data from e-Government services as input requires test data that is real-life-like but where the privacy of personal information is guaranteed. Applications with such strong requirement include information exchange between countries, medicine, banking, etc. This review aims to synthesize the current state-of-the-practice in this domain. Objectives: The objective of this Systematic Review is to identify existing approaches for creating and evolving synthetic test data without using real-life raw data. Methods: We followed well-known methodologies for conducting systematic literature reviews, including the ones from Kitchenham as well as guidelines for analysing the limitations of our review and its threats to validity. Results: A variety of methods and tools exist for creating privacy-preserving test data. Our search found 1,013 publications in IEEE Xplore, ACM Digital Library, and SCOPUS. We extracted data from 75 of those publications and identified 37 approaches that answer our research question partly. A common prerequisite for using these methods and tools is direct access to real-life data for data anonymization or synthetic test data generation. Nine existing synthetic test data generation approaches were identified that were closest to answering our research question. Nevertheless, further work would be needed to add the ability to evolve synthetic test data to the existing approaches. Conclusions: None of the publications really covered our requirements completely, only partially. Synthetic test data evolution is a field that has not received much attention from researchers but needs to be explored in Digital Government Solutions, especially since new legal regulations are being placed in force in many countries.

Figures (9)

• Figure 1: Centralized vs. Decentralized DGS
• Figure 2: Compatibility of test data among different government entities in a decentralized DGS
• Figure 3: Results of the Title and Abstract Analysis and the Full Text Analysis
• Figure 4: Type of approach - output data
• Figure 5: Type of approach - input data