DBOS Network Sensing: A Web Services Approach to Collaborative Awareness

TL;DR

This work addresses the need for real-time, cross-node network awareness to bolster resilience and security in distributed web services. It introduces DBOS with GraphBLAS hypersparse traffic matrices to enable collaborative analytics, implementing two local aggregation workflows (Python-GraphBLAS and OneSparse PostgreSQL) and two global coordination approaches (pPython and shared file system). Results from MIT SuperCloud benchmarks demonstrate sustained high request rates ($>10^5$ RPS) and linear scaling up to 64 nodes, with Python-based workflows achieving higher throughput than PostgreSQL due to lower transactional overhead. The study provides a practical, embedded pathway for integrating collaborative network analysis into web service runtimes, highlighting scalability limits and future opportunities for anonymization, real-world deployment, and coordinated response.

Abstract

DBOS (DataBase Operating System) is a novel capability that integrates web services, operating system functions, and database features to significantly reduce web-deployment effort while increasing resilience. Integration of high performance network sensing enables DBOS web services to collaboratively create a shared awareness of their network environments to enhance their collective resilience and security. Network sensing is added to DBOS using GraphBLAS hypersparse traffic matrices via two approaches: (1) Python-GraphBLAS and (2) OneSparse PostgreSQL. These capabilities are demonstrated using the workflow and analytics from the IEEE/MIT/Amazon Anonymized Network Sensing Graph Challenge. The system was parallelized using pPython and benchmarked using 64 compute nodes on the MIT SuperCloud. The web request rate sustained by a single DBOS instance was ${>}10^5$, well above the required maximum, indicating that network sensing can be added to DBOS with negligible overhead. For collaborative awareness, many DBOS instances were connected to a single DBOS aggregator. The Python-GraphBLAS and OneSparse PostgreSQL implementations scaled linearly up to 64 and 32 nodes respectively. These results suggest that DBOS collaborative network awareness can be achieved with a negligible increase in computing resources.

Figures (10)

• Figure 1: Collaborative Awareness. This approach enhances both the efficiency and precision of threat detection, allowing systems to identify attacks that would likely go unnoticed by isolated monitoring setups badsha2019privacylocasto2005towardskatti2005collaborating. Stovepipe isolation (left) prevents lateral communication between organizational silos, maximizing privacy but limiting visibility. Radical transparency (right) mandates universal disclosure of all security-relevant data, theoretically maximizing defensive capabilities but proving infeasible due to data sharing restrictions. Collaborative awareness (center) enables exchange of sufficient data, providing actionable intelligence while maintaining the highest regard for privacy.
• Figure 2: DBOS Collaborative Awareness Architecture. Integration of GraphBLAS hypersparse traffic matrix construction into DBOS enables DBOS web services to collaboratively create a shared awareness of their network environments to enhance their collective resilience and security.
• Figure 3: Overview of Local and Global Workflows. Each DBOS instance constructs base GraphBLAS traffic matrices from observed source and destination IP pairs. These are aggregated into a local GraphBLAS traffic matrix, which is then transmitted to a coordinator. The coordinator aggregates the local traffic matrices into a global GraphBLAS traffic matrix, producing a system-wide summary of network activity.
• Figure 4: Python Local Aggregation Workflow
• Figure 5: PostgreSQL Local Aggregation Workflow