Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bring Your Own Objective: Inter-operability of Network Objectives in Datacenters

Sanjoli Narang, Anup Agarwal, Venkat Arun, Manya Ghobadi

TL;DR

DMart addresses the limitations of single-objective datacenter fabrics by introducing a market-based, RTT-scale mechanism that treats bandwidth as a competitive resource. Flows encode objective-specific urgency into per-round bids, while switch-local, per-link second-price auctions coordinate end-to-end transmission without centralized optimization. The design supports multiple objectives (FCT, deadlines, coflows, fairness) through a unified bidding API and convergence guarantees under price-taking behavior. Empirical evaluation on packet-level ns-3 simulations shows DMart matches specialized schedulers on their home metrics and delivers substantial reductions in deadline misses and coflow completion times, while remaining robust to workload drift and mixed-objective traffic. This approach offers a scalable, future-proof path to diverse datacenter performance goals.

Abstract

Datacenter networks are currently locked in a "tyranny of the single objective". While modern workloads demand diverse performance goals, ranging from coflow completion times, per-flow fairness, short-flow latencies, existing fabrics are typically hardcoded for a single metric. This rigid coupling ensures peak performance when application and network objectives align, but results in abysmal performance when they diverge. We propose DMart, a decentralized scheduling framework that treats network bandwidth as a competitive marketplace. In DMart, applications independently encode the urgency and importance of their network traffic into autonomous bids, allowing diverse objectives to co-exist natively on the same fabric. To meet the extreme scale and sub-microsecond requirements of modern datacenters, DMart implements distributed, per-link, per-RTT auctions, without relying on ILPs, centralized schedulers, or complex priority queues. We evaluate DMart using packet-level simulations and compare it against network schedulers designed for individual metrics, e.g., pFabric and Sincronia. DMart matches the performance of specialized schedulers on their own "home turf" while simultaneously optimizing secondary metrics. Compared to pFabric and Sincronia, DMart reduces deadline misses by 2x and coflow completion times by 1.6x respectively, while matching pFabric short-flow completion times.

Bring Your Own Objective: Inter-operability of Network Objectives in Datacenters

TL;DR

DMart addresses the limitations of single-objective datacenter fabrics by introducing a market-based, RTT-scale mechanism that treats bandwidth as a competitive resource. Flows encode objective-specific urgency into per-round bids, while switch-local, per-link second-price auctions coordinate end-to-end transmission without centralized optimization. The design supports multiple objectives (FCT, deadlines, coflows, fairness) through a unified bidding API and convergence guarantees under price-taking behavior. Empirical evaluation on packet-level ns-3 simulations shows DMart matches specialized schedulers on their home metrics and delivers substantial reductions in deadline misses and coflow completion times, while remaining robust to workload drift and mixed-objective traffic. This approach offers a scalable, future-proof path to diverse datacenter performance goals.

Abstract

Datacenter networks are currently locked in a "tyranny of the single objective". While modern workloads demand diverse performance goals, ranging from coflow completion times, per-flow fairness, short-flow latencies, existing fabrics are typically hardcoded for a single metric. This rigid coupling ensures peak performance when application and network objectives align, but results in abysmal performance when they diverge. We propose DMart, a decentralized scheduling framework that treats network bandwidth as a competitive marketplace. In DMart, applications independently encode the urgency and importance of their network traffic into autonomous bids, allowing diverse objectives to co-exist natively on the same fabric. To meet the extreme scale and sub-microsecond requirements of modern datacenters, DMart implements distributed, per-link, per-RTT auctions, without relying on ILPs, centralized schedulers, or complex priority queues. We evaluate DMart using packet-level simulations and compare it against network schedulers designed for individual metrics, e.g., pFabric and Sincronia. DMart matches the performance of specialized schedulers on their own "home turf" while simultaneously optimizing secondary metrics. Compared to pFabric and Sincronia, DMart reduces deadline misses by 2x and coflow completion times by 1.6x respectively, while matching pFabric short-flow completion times.
Paper Structure (24 sections, 2 theorems, 28 equations, 17 figures, 2 tables)

This paper contains 24 sections, 2 theorems, 28 equations, 17 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Related Work
  4. Design Overview
  5. Auction-based Link Allocation
  6. Challenges in Market Design
  7. Design Rationale and Theory
  8. Incentives in Second-Price Auctions
  9. Objectives and Bidding Agents
  10. Common Datacenter Objectives
  11. Convergence under Learning Agents
  12. Implementation Deep Dive
  13. Packet Metadata for Bidding
  14. Rate Control and Queuing
  15. Evaluation
  16. ...and 9 more sections

Key Result

theorem 1

If the market is price-taking, as in Equation eq:exo, then bidding truthful marginal utility $b_i(S_t)=\Delta U_i(S_t)$ in each round is a weakly dominant strategy.

Figures (17)

  • Figure 1: Single-link example with heterogeneous SLOs (F1 slowdown, F2 completion time, F3 minimum bandwidth). Outcomes: (a) SRPT (67%, 2 units, 0%); (b) Round Robin (233%, 5 units, 0%); (c) Fairness (267%, 6 units, 33%); (d) Proportional(5:4:1) (187%, 5 units, 10%); (e) SLO-optimal (11%, 5.55 units, 10%).
  • Figure 2: Illustration of end-to-end flow control using independent per-link second-price auctions on a 2-layer topology.
  • Figure 3: Link idling and deadlock in link auctions due to dependencies. Overcommitment increases winners quota at $S_1$ to unblock.
  • Figure 4: Deadline-objective bidding as a function of state. With a higher budget ($C=4000$), bidders remain active over a larger region of the state space than in the lower-budget case ($C=2000$).
  • Figure 5: TCP Packet with the new Market Header
  • ...and 12 more figures

Theorems & Definitions (4)

  • theorem 1: Truthful Bidding under Price-Taking Environment
  • proof
  • theorem 2: Myopic Truthfulness under Price-Taking Behaviour
  • proof