Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inference

TL;DR

Nesa presents a model-agnostic hybrid sharding framework (BSNS) for decentralized AI inference that distributes model shards across a swarm of heterogeneous nodes via blockchain-based sequential sharding. By blending topology-aware node selection, persistent homology, and dynamic rebalancing with compression (dynamic quantization, mixed matrix decomposition) and PEFT adapters, the approach enables scalable inference and training on consumer hardware while preserving data privacy through TEEs, ZKML, CDV, and Split Learning. Key contributions include the BSNS sharding protocol, dynamic MTPP-based graph partitioning, topology-aware routing, and a hardware-software co-optimization privacy stack. The results indicate minimal accuracy loss under compression and demonstrate practical throughput improvements and secure data handling, highlighting potential for broad AI democratization. Overall, the paper advances decentralized AI by combining model-agnostic sharding, topology-guided routing, and privacy-preserving protocols to enable secure, efficient inference on a distributed network.

Abstract

The rapid growth of large-scale AI models, particularly large language models has brought significant challenges in data privacy, computational resources, and accessibility. Traditional centralized architectures often struggle to meet required data security and scalability needs which hinders the democratization of AI systems. Nesa introduces a model-agnostic sharding framework designed for decentralized AI inference. Our framework uses blockchain-based sequential deep neural network sharding to distribute computational tasks across a diverse network of nodes based on a personalised heuristic and routing mechanism. This enables efficient distributed training and inference for recent large-scale models even on consumer-grade hardware. We use compression techniques like dynamic blockwise quantization and mixed matrix decomposition to reduce data transfer and memory needs. We also integrate robust security measures, including hardware-based trusted execution environments to ensure data integrity and confidentiality. Evaluating our system across various natural language processing and vision tasks shows that these compression strategies do not compromise model accuracy. Our results highlight the potential to democratize access to cutting-edge AI technologies by enabling secure and efficient inference on a decentralized network.

Figures (5)

• Figure 1: High-level overview of the Nesa network.
• Figure 2: Orchestrator is chosen from the pool of accessible nodes and picks nodes to build a whole set of blocks (e.g., transformer blocks) according to the model architecture that needs to be run. To enable direct routing between blocks, each node has a unique section of the computational graph. Initiating from the orchestrator, the inference is routed to the first block and then through each remaining block directly in order.
• Figure 3: Toy example for simplicial homology: the left simplicial complex consists of three vertices (0-simplices) and three edges (1-simplices) whereas the right consists of four vertices and four edges. Topologically, these two complexes are identical from the viewpoint of simplicial homology, but they are distinguishable via persistent homology.
• Figure 4: Visualizing persistent homology for a simple graph of 3 vertices on an equilateral triangle: the top row illustrates 3 snapshots of the filtration sequence of three circles of growing radii (filtration parameter). The bottom row demonstrates the respective persistence diagrams. Note that in $H_0(M)$, the initial 3 dots are shifted slightly to aid the eye but they would overlap precisely in reality.
• Figure 5: Overview of genetic algorithms for optimization and a decision-making framework based on multi-objective optimization metrics for selecting nodes in a network.