Retrofitting Service Dependency Discovery in Distributed Systems

TL;DR

This work tackles the challenge of rebuilding accurate service dependency graphs in NAT-rich distributed systems. It proposes XXXX, a run-time, non-intrusive instrumentation framework that piggybacks a small identifier into TCP header options via eBPF to map communications to specific userspace processes, even through NAT. The approach is protocol-agnostic, supports encrypted TCP-based protocols, and supports incremental deployment with minimal disruption. Empirical evaluation across NAT-free, internal NAT, and external NAT configurations on three microservice benchmarks shows high precision and recall, with low runtime overhead and better robustness to NAT than state-of-the-art baselines. The work enables scalable root-cause analysis in complex deployments, while acknowledging TCP-only scope as a limitation and outlining UDP support as future work.

Abstract

Modern distributed systems rely on complex networks of interconnected services, creating direct or indirect dependencies that can propagate faults and cause cascading failures. To localize the root cause of performance degradation in these environments, constructing a service dependency graph is highly beneficial. However, building an accurate service dependency graph is impaired by complex routing techniques, such as Network Address Translation (NAT), an essential mechanism for connecting services across networks. NAT obfuscates the actual hosts running the services, causing existing run-time approaches that passively observe network metadata to fail in accurately inferring service dependencies. To this end, this paper introduces XXXX, a novel run-time system for constructing process-level service dependency graphs. It operates without source code instrumentation and remains resilient under complex network routing mechanisms, including NAT. XXXX implements a non-disruptive method of injecting metadata onto a TCP packet's header that maintains protocol correctness across host boundaries. In other words, if no receiving agent is present, the instrumentation leaves existing TCP connections unaffected, ensuring non-disruptive operation when it is partially deployed across hosts. We evaluated XXXX extensively against three state-of-the-art systems across nine scenarios, involving three network configurations (NAT-free, internal-NAT, external-NAT) and three microservice benchmarks. XXXX was the only approach that performed consistently across networking configurations. With regards to correctness, it performed on par with, or better than, the state-of-the-art with precision and recall values of 100% in the majority of the scenarios.

Figures (9)

• Figure 1: NAT-free network configuration. Services A and B run on separate hosts within the same network and communicate over a direct connection.
• Figure 2: Internal NAT network configuration. Services A and B run in containers on separate hosts, with NAT applied at each host boundary.
• Figure 3: External NAT network configuration. Services A and B run on hosts in separate LANs, with NAT applied at each LAN's boundary.
• Figure 4: Overview of XXXX’s discovery process. A packet transmitted through a connection associated with socket S1, owned by a target process P1, triggers the execution of the egress eBPF programs A and B, which determine whether the packet requires instrumentation. If so, program C injects an identifier ID into the packet and returns it to the networking stack for normal processing. On ingress, program D detects the identifier and records the receiving socket S2, while program E links the socket to the receiving process P2, thereby completing service discovery for that connection.
• Figure 5: XXXX service map reconstruction for the NAT-free network configuration. (a) Online Boutique online-boutique; (b) Media gan2019open; (c) Social Media gan2019open.