Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A System Development Kit for Big Data Applications on FPGA-based Clusters: The EVEREST Approach

Christian Pilato, Subhadeep Banik, Jakub Beranek, Fabien Brocheton, Jeronimo Castrillon, Riccardo Cevasco, Radim Cmar, Serena Curzel, Fabrizio Ferrandi, Karl F. A. Friebel, Antonella Galizia, Matteo Grasso, Paulo Silva, Jan Martinovic, Gianluca Palermo, Michele Paolino, Andrea Parodi, Antonio Parodi, Fabio Pintus, Raphael Polig, David Poulet, Francesco Regazzoni, Burkhard Ringlein, Roberto Rocco, Katerina Slaninova, Tom Slooff, Stephanie Soldavini, Felix Suchert, Mattia Tibaldi, Beat Weiss, Christoph Hagleitner

TL;DR

The paper addresses the challenge of developing energy-efficient FPGA-accelerated big data workloads in data-center environments. It presents the EVEREST SDK, a unified framework combining data-driven compilation, a virtualized runtime, and an anomaly-detection service, built on MLIR-based representations and ConDRust coordination. Key contributions include the data-driven compilation framework, system-level generation via Olympus, a SR-IOV-enabled virtualization layer, a dynamic autotuning component (mARGOt), and an anomaly-detection service, demonstrated across weather, renewable-energy, air-quality, and traffic use cases. The approach enables efficient, adaptable FPGA deployments and improved designer productivity for complex data-centric workloads in heterogeneous hardware settings.

Abstract

Modern big data workflows are characterized by computationally intensive kernels. The simulated results are often combined with knowledge extracted from AI models to ultimately support decision-making. These energy-hungry workflows are increasingly executed in data centers with energy-efficient hardware accelerators since FPGAs are well-suited for this task due to their inherent parallelism. We present the H2020 project EVEREST, which has developed a system development kit (SDK) to simplify the creation of FPGA-accelerated kernels and manage the execution at runtime through a virtualization environment. This paper describes the main components of the EVEREST SDK and the benefits that can be achieved in our use cases.

A System Development Kit for Big Data Applications on FPGA-based Clusters: The EVEREST Approach

TL;DR

The paper addresses the challenge of developing energy-efficient FPGA-accelerated big data workloads in data-center environments. It presents the EVEREST SDK, a unified framework combining data-driven compilation, a virtualized runtime, and an anomaly-detection service, built on MLIR-based representations and ConDRust coordination. Key contributions include the data-driven compilation framework, system-level generation via Olympus, a SR-IOV-enabled virtualization layer, a dynamic autotuning component (mARGOt), and an anomaly-detection service, demonstrated across weather, renewable-energy, air-quality, and traffic use cases. The approach enables efficient, adaptable FPGA deployments and improved designer productivity for complex data-centric workloads in heterogeneous hardware settings.

Abstract

Modern big data workflows are characterized by computationally intensive kernels. The simulated results are often combined with knowledge extracted from AI models to ultimately support decision-making. These energy-hungry workflows are increasingly executed in data centers with energy-efficient hardware accelerators since FPGAs are well-suited for this task due to their inherent parallelism. We present the H2020 project EVEREST, which has developed a system development kit (SDK) to simplify the creation of FPGA-accelerated kernels and manage the execution at runtime through a virtualization environment. This paper describes the main components of the EVEREST SDK and the benefits that can be achieved in our use cases.
Paper Structure (21 sections, 1 equation, 6 figures)

This paper contains 21 sections, 1 equation, 6 figures.

Table of Contents

  1. Introduction
  2. Application Use Cases
  3. WRF-based Weather Simulations
  4. Renewable-energy prediction
  5. Air-quality monitoring
  6. Traffic modeling
  7. EVEREST Target Systems
  8. EVEREST System Development Kit
  9. Data-driven Compilation Framework
  10. Input languages and abstractions
  11. EVEREST kernel language
  12. EVEREST coordination language
  13. MLIR representations
  14. System-level generation
  15. Virtualized Runtime Environment
  16. ...and 6 more sections

Figures (6)

  • Figure 1: The EVEREST converged heterogeneous platform
  • Figure 2: EVEREST SDK components.
  • Figure 3: Example of major absorber in the EVEREST Kernel Language.
  • Figure 4: Example ConDRust syntax for map matching a single element.
  • Figure 5: EVEREST MLIR dialects (in blue) and their integration with other core MLIR dialects (in green). Dialects under construction in grey-blue
  • ...and 1 more figures