Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deadlock-free routing for Full-mesh networks without using Virtual Channels

Alejandro Cano, Cristóbal Camarero, Carmen Martínez, Ramón Beivide

TL;DR

The paper tackles deadlock-free non-minimal routing in full-mesh, high-radix networks without virtual channels. It introduces TERA, a topology-embedded routing algorithm that partitions the FM into a service topology with deadlock-free minimal routing and a main topology for non-minimal routing, ensuring deadlock-free and livelock-free operation with bounded hops. Compared to VC-based adaptive routings and VC-less link-ordering, TERA delivers substantial throughput gains under adversarial and application workloads, while significantly reducing buffer requirements. Early 2D-HyperX results indicate TERA can outperform VC-based peers using the same number of VCs by up to 32%, highlighting its practicality for scalable interconnects in data centers and supercomputers.

Abstract

High-radix, low-diameter networks like HyperX and Dragonfly use a Full-mesh core, and rely on multiple virtual channels (VCs) to avoid packet deadlocks in adaptive routing. However, VCs introduce significant overhead in the switch in terms of area, power, and design complexity, limiting the switch scalability. This paper starts by revisiting VC-less routing through link ordering schemes in Full-mesh networks, which offer implementation simplicity but suffer from performance degradation under adversarial traffic. Thus, to overcome these challenges, we propose TERA (Topology-Embedded Routing Algorithm), a novel routing algorithm which employs an embedded physical subnetwork to provide deadlock-free non-minimal paths without using VCs. In a Full-mesh network, TERA outperforms link ordering routing algorithms by 80% when dealing with adversarial traffic, and up to 100% in application kernels. Furthermore, compared to other VC-based approaches, it reduces buffer requirements by 50%, while maintaining comparable latency and throughput. Lastly, early results from a 2D-HyperX evaluation show that TERA outperforms state-of-the-art algorithms that use the same number of VCs, achieving performance improvements of up to 32%.

Deadlock-free routing for Full-mesh networks without using Virtual Channels

TL;DR

The paper tackles deadlock-free non-minimal routing in full-mesh, high-radix networks without virtual channels. It introduces TERA, a topology-embedded routing algorithm that partitions the FM into a service topology with deadlock-free minimal routing and a main topology for non-minimal routing, ensuring deadlock-free and livelock-free operation with bounded hops. Compared to VC-based adaptive routings and VC-less link-ordering, TERA delivers substantial throughput gains under adversarial and application workloads, while significantly reducing buffer requirements. Early 2D-HyperX results indicate TERA can outperform VC-based peers using the same number of VCs by up to 32%, highlighting its practicality for scalable interconnects in data centers and supercomputers.

Abstract

High-radix, low-diameter networks like HyperX and Dragonfly use a Full-mesh core, and rely on multiple virtual channels (VCs) to avoid packet deadlocks in adaptive routing. However, VCs introduce significant overhead in the switch in terms of area, power, and design complexity, limiting the switch scalability. This paper starts by revisiting VC-less routing through link ordering schemes in Full-mesh networks, which offer implementation simplicity but suffer from performance degradation under adversarial traffic. Thus, to overcome these challenges, we propose TERA (Topology-Embedded Routing Algorithm), a novel routing algorithm which employs an embedded physical subnetwork to provide deadlock-free non-minimal paths without using VCs. In a Full-mesh network, TERA outperforms link ordering routing algorithms by 80% when dealing with adversarial traffic, and up to 100% in application kernels. Furthermore, compared to other VC-based approaches, it reduces buffer requirements by 50%, while maintaining comparable latency and throughput. Lastly, early results from a 2D-HyperX evaluation show that TERA outperforms state-of-the-art algorithms that use the same number of VCs, achieving performance improvements of up to 32%.
Paper Structure (19 sections, 1 theorem, 6 equations, 10 figures, 1 table, 1 algorithm)

This paper contains 19 sections, 1 theorem, 6 equations, 10 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation
  3. Buffering resources
  4. Buffer Size
  5. Buffer Utilization Inefficiency
  6. High-radix low-diameter networks
  7. Link ordering Schemes without VCs
  8. The Topology Embedded Routing Algorithm
  9. Evaluation of service topologies
  10. Methodology
  11. Performance Evaluation
  12. Link ordering evaluation
  13. Service topology selection
  14. Bernoulli traffic evaluation
  15. Application results
  16. ...and 4 more sections

Key Result

Theorem 3.2

If a routing scheme for a FM is based on an ordering of the arcs and ensures that all links can be used by the same number of source/destination pairs, then the number of allowed paths of length 2 is $\frac{1}{2}n(n-1)(n-2)$.

Figures (10)

  • Figure 1: Full-mesh network with 16 servers: 4 switches with 4 servers per switch.
  • Figure 2: K$_4$ with sRINR labelling.
  • Figure 3: $K_4$ with a Path, and a Hypercube embedded.
  • Figure 4: Estimated throughput for different service topologies.
  • Figure 5: Time to finish of different traffic patterns in a FM$_{64}$ with 4096 servers.
  • ...and 5 more figures

Theorems & Definitions (7)

  • Definition 3.1
  • Theorem 3.2
  • Definition 3.3
  • Claim 3.4
  • Definition 4.1: Main and service topologies
  • proof : Proof of Theorem \ref{['claim:utilizationfairpaths']}
  • proof : Proof of Claim \ref{['claim:intermediates_forward']}