Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autonomy Loops for Monitoring, Operational Data Analytics, Feedback, and Response in HPC Operations

Francieli Boito, Jim Brandt, Valeria Cardellini, Philip Carns, Florina M. Ciorba, Hilary Egan, Ahmed Eleliemy, Ann Gentile, Thomas Gruber, Jeff Hanson, Utz-Uwe Haus, Kevin Huck, Thomas Ilsche, Thomas Jakobsche, Terry Jones, Sven Karlsson, Abdullah Mueen, Michael Ott, Tapasya Patki, Ivy Peng, Krishnan Raghavan, Stephen Simms, Kathleen Shoga, Michael Showerman, Devesh Tiwari, Torsten Wilde, Keiji Yamamoto

TL;DR

The paper addresses the need for scalable MODA in HPC operations and the shortcomings of human-in-the-loop approaches. It advocates using the MAPE-K autonomic computing pattern to define MODA autonomy loops and to derive interoperable interfaces and telemetry hooks. The authors present an initial set of use cases, notably Scheduler, and detail design considerations, data requirements, and extensibility to other cases. The work lays groundwork for community-driven conventions, testbeds such as OpenCUBE, and open datasets to enable pervasive deployment of autonomous feedback and response in HPC.

Abstract

Many High Performance Computing (HPC) facilities have developed and deployed frameworks in support of continuous monitoring and operational data analytics (MODA) to help improve efficiency and throughput. Because of the complexity and scale of systems and workflows and the need for low-latency response to address dynamic circumstances, automated feedback and response have the potential to be more effective than current human-in-the-loop approaches which are laborious and error prone. Progress has been limited, however, by factors such as the lack of infrastructure and feedback hooks, and successful deployment is often site- and case-specific. In this position paper we report on the outcomes and plans from a recent Dagstuhl Seminar, seeking to carve a path for community progress in the development of autonomous feedback loops for MODA, based on the established formalism of similar (MAPE-K) loops in autonomous computing and self-adaptive systems. By defining and developing such loops for significant cases experienced across HPC sites, we seek to extract commonalities and develop conventions that will facilitate interoperability and interchangeability with system hardware, software, and applications across different sites, and will motivate vendors and others to provide telemetry interfaces and feedback hooks to enable community development and pervasive deployment of MODA autonomy loops.

Autonomy Loops for Monitoring, Operational Data Analytics, Feedback, and Response in HPC Operations

TL;DR

The paper addresses the need for scalable MODA in HPC operations and the shortcomings of human-in-the-loop approaches. It advocates using the MAPE-K autonomic computing pattern to define MODA autonomy loops and to derive interoperable interfaces and telemetry hooks. The authors present an initial set of use cases, notably Scheduler, and detail design considerations, data requirements, and extensibility to other cases. The work lays groundwork for community-driven conventions, testbeds such as OpenCUBE, and open datasets to enable pervasive deployment of autonomous feedback and response in HPC.

Abstract

Many High Performance Computing (HPC) facilities have developed and deployed frameworks in support of continuous monitoring and operational data analytics (MODA) to help improve efficiency and throughput. Because of the complexity and scale of systems and workflows and the need for low-latency response to address dynamic circumstances, automated feedback and response have the potential to be more effective than current human-in-the-loop approaches which are laborious and error prone. Progress has been limited, however, by factors such as the lack of infrastructure and feedback hooks, and successful deployment is often site- and case-specific. In this position paper we report on the outcomes and plans from a recent Dagstuhl Seminar, seeking to carve a path for community progress in the development of autonomous feedback loops for MODA, based on the established formalism of similar (MAPE-K) loops in autonomous computing and self-adaptive systems. By defining and developing such loops for significant cases experienced across HPC sites, we seek to extract commonalities and develop conventions that will facilitate interoperability and interchangeability with system hardware, software, and applications across different sites, and will motivate vendors and others to provide telemetry interfaces and feedback hooks to enable community development and pervasive deployment of MODA autonomy loops.
Paper Structure (5 sections, 3 figures)

This paper contains 5 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Autonomy Loop Case Study Methodology
  3. Driving Generalization Through Initial Case
  4. Design Changes for Autonomous MODA
  5. Conclusion

Figures (3)

  • Figure 1: Vision of holistic monitoring and operational data analytics.
  • Figure 2: Design Patterns for MAPE-K loops. Leveraging the MAPE-K formalism will facilitate application of the designs to MODA autonomy loops.
  • Figure 3: Scheduler use case and its MAPE-K loop components.