Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Design Guidelines for Apache Kafka Driven Data Management and Distribution in Smart Cities

Theofanis P. Raptis, Claudio Cicconetti, Manolis Falelakis, Tassos Kanellos, Tomás Pariente Lobo

TL;DR

A modular architecture is proposed to leverage innovative technologies for data acquisition, management and distribution, develop a multi-layer engineering solution for revealing valuable and hidden societal knowledge in smart cities environment, and tackle the main issues in tasks involving complex data flows and provide general guidelines to solve them.

Abstract

Smart city management is going through a remarkable transition, in terms of quality and diversity of services provided to the end-users. The stakeholders that deliver pervasive applications are now able to address fundamental challenges in the big data value chain, from data acquisition, data analysis and processing, data storage and curation, and data visualisation in real scenarios. Industry 4.0 is pushing this trend forward, demanding for servitization of products and data, also for the smart cities sector where humans, sensors and devices are operating in strict collaboration. The data produced by the ubiquitous devices must be processed quickly to allow the implementation of reactive services such as situational awareness, video surveillance and geo-localization, while always ensuring the safety and privacy of involved citizens. This paper proposes a modular architecture to (i) leverage innovative technologies for data acquisition, management and distribution (such as Apache Kafka and Apache NiFi), (ii) develop a multi-layer engineering solution for revealing valuable and hidden societal knowledge in smart cities environment, and (iii) tackle the main issues in tasks involving complex data flows and provide general guidelines to solve them. We derived some guidelines from an experimental setting performed together with leading industrial technical departments to accomplish an efficient system for monitoring and servitization of smart city assets, with a scalable platform that confirms its usefulness in numerous smart city use cases with different needs.

Design Guidelines for Apache Kafka Driven Data Management and Distribution in Smart Cities

TL;DR

A modular architecture is proposed to leverage innovative technologies for data acquisition, management and distribution, develop a multi-layer engineering solution for revealing valuable and hidden societal knowledge in smart cities environment, and tackle the main issues in tasks involving complex data flows and provide general guidelines to solve them.

Abstract

Smart city management is going through a remarkable transition, in terms of quality and diversity of services provided to the end-users. The stakeholders that deliver pervasive applications are now able to address fundamental challenges in the big data value chain, from data acquisition, data analysis and processing, data storage and curation, and data visualisation in real scenarios. Industry 4.0 is pushing this trend forward, demanding for servitization of products and data, also for the smart cities sector where humans, sensors and devices are operating in strict collaboration. The data produced by the ubiquitous devices must be processed quickly to allow the implementation of reactive services such as situational awareness, video surveillance and geo-localization, while always ensuring the safety and privacy of involved citizens. This paper proposes a modular architecture to (i) leverage innovative technologies for data acquisition, management and distribution (such as Apache Kafka and Apache NiFi), (ii) develop a multi-layer engineering solution for revealing valuable and hidden societal knowledge in smart cities environment, and (iii) tackle the main issues in tasks involving complex data flows and provide general guidelines to solve them. We derived some guidelines from an experimental setting performed together with leading industrial technical departments to accomplish an efficient system for monitoring and servitization of smart city assets, with a scalable platform that confirms its usefulness in numerous smart city use cases with different needs.
Paper Structure (15 sections, 4 figures, 2 tables)

This paper contains 15 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related work
  4. DMP Software Architecture Design
  5. DatAna
  6. DFB
  7. StreamHandler
  8. HDD
  9. AV Data
  10. Evaluation of the setup
  11. DatAna
  12. DFB
  13. StreamHandler
  14. HDD
  15. Design guidelines

Figures (4)

  • Figure 1: Transition of smart city data management and distribution systems from vertical silos to horizontal (cloud-based) platforms and positioning of the https://www.marvel-project.eu/'s proposition for multi-layer E2F2C environments.
  • Figure 2: E2F2C view of the https://www.marvel-project.eu/'s pilots.
  • Figure 3: Schematic of a typical multi-modal, i.e., audio + video, real-time analytics application in https://www.marvel-project.eu/. The mapping of components to the edge/fog/cloud layers is only indicative: in real applications this may depends on physical, environmental, and administrative constraints.
  • Figure 4: Results of the simulated measurements for HDD component.