Model-driven development of data intensive applications over cloud resources

TL;DR

The paper tackles the challenge of designing data-intensive streaming applications in cloud environments with real-time and cost constraints. It proposes a model-driven, stepwise refinement methodology that centers on Petri Net models to capture both functional behavior and non-functional properties, enabling formal analysis, simulation, and approximate performance/economic evaluation across cloud mappings. Through the Wavefront case study, it demonstrates modular functional/operational modeling, cloud-resource linking, and both grid and pipeline deployment paradigms, along with quantitative and qualitative analyses (including stochastic PN methods and LP-based bounds). The work advances a cohesive framework for early-resource-aware design, performance prediction, and autonomic cloud provisioning, with Langliers providing a high-level executable specification language to support the methodology. Practically, this approach supports what-if exploration of mappings, resource configurations, and QoS/cost trade-offs, potentially improving SLA adherence and cost efficiency in cloud-based streaming pipelines.

Abstract

The proliferation of sensors over the last years has generated large amounts of raw data, forming data streams that need to be processed. In many cases, cloud resources are used for such processing, exploiting their flexibility, but these sensor streaming applications often need to support operational and control actions that have real-time and low-latency requirements that go beyond the cost effective and flexible solutions supported by existing cloud frameworks, such as Apache Kafka, Apache Spark Streaming, or Map-Reduce Streams. In this paper, we describe a model-driven and stepwise refinement methodological approach for streaming applications executed over clouds. The central role is assigned to a set of Petri Net models for specifying functional and non-functional requirements. They support model reuse, and a way to combine formal analysis, simulation, and approximate computation of minimal and maximal boundaries of non-functional requirements when the problem is either mathematically or computationally intractable. We show how our proposal can assist developers in their design and implementation decisions from a performance perspective. Our methodology allows to conduct performance analysis: The methodology is intended for all the engineering process stages, and we can (i) analyse how it can be mapped onto cloud resources, and (ii) obtain key performance indicators, including throughput or economic cost, so that developers are assisted in their development tasks and in their decision taking. In order to illustrate our approach, we make use of the pipelined wavefront array.

Figures (19)

• Figure 1: Life cycle and management methodology for CDFAs over cloud platforms.
• Figure 2: Wavefront processing of $Z^{(k)} = Y^{(k)} + A \cdot X^{(k)}$, $k=1,2,...$.
• Figure 5: Refinement of the Functional Petri Net model to take into account the operational data transmission process that must be shared for three functional data transmission processes
• Figure 6: Basic modules for the construction of the functional Petri Net model of the wavefront algorithm: (a) Computational Process associated to a node; (b) Data Transmission Process for the external/internal data streams.
• Figure 7: Untimed functional Petri Net model of the wavefront algorithm