MDTP -- An Adaptive Multi-Source Data Transfer Protocol

TL;DR

MDTP addresses the challenge of transferring large scientific datasets by leveraging multiple replicas through adaptive, bin-packing-based chunking. The protocol dynamically allocates variable-sized chunks to heterogeneous servers based on observed throughput, aiming to keep all concurrent transfers finishing in roughly the same time and to utilize all replicas fairly. A Python prototype using HTTP byte-range requests demonstrates that MDTP outperforms Aria2, BitTorrent, and static chunking under diverse conditions, with transfer-time savings up to about 22% over Aria2 and improved replica utilization. The work offers practical implications for high-volume data transfers in scientific communities, enabling faster, more balanced data dissemination across distributed infrastructures.

Abstract

Scientific data volume is growing in size, and as a direct result, the need for faster transfers is also increasing. The scientific community has sought to leverage parallel transfer methods using multi-threaded and multi-source download models to reduce download times. In multi-source transfers, a client downloads data from multiple replicated servers in parallel. Tools such as Aria2 and BitTorrent support such multi-source transfers and have shown improved transfer times. In this work, we introduce Multi-Source Data Transfer Protocol, MDTP, which further improves multi-source transfer performance. MDTP logically divides a file request into smaller chunk requests and distributes the chunk requests across multiple servers. Chunk sizes are adapted based on each server's performance but selected in a way that ensures each round of requests completes around the same time. We formulate this chunk-size allocation problem as a variant of the bin-packing problem, where adaptive chunking efficiently fills the available capacity "bins" corresponding to each server. Our evaluation shows that MDTP reduces transfer times by 10-22% compared to Aria2, the fastest alternative. Comparisons with other protocols, such as static chunking and BitTorrent, demonstrate even greater improvements. Additionally, we show that MDTP distributes load proportionally across all available replicas, not just the fastest ones, which improves throughput. Finally, we show MDTP maintains high throughput even when latency increases or bandwidth to the fastest server decreases.

Figures (5)

• Figure 1: Figure showing variable sized chunks for different servers with different speeds to finish in same time in MDTP
• Figure 2: (a) Average transfer time trend including disk write delay. MDTP, Aria2, and Static Chunking perform similarly while BitTorrent is much slower, (b) Average transfer time trends without disk write delay. We omit BitTorrent due to its lower performance. MDTP performs better than Aria2 and Static Chunking for larger file sizes, and (c) Number of peers seeding a 2GB file in different download iterations. The unpredictable number of seeders make BitTorrent unsuitable for this use case.
• Figure 3: (a) Average transfer times with and without 0.5s added latency for a 64GB file, including disk write delay. MDTP and Aria2 exhibit similar behavior, with Aria2 being more inconsistent, while Static Chunking has higher latency. (b) Transfer times with and without 0.5s added latency for a 64GB file, excluding disk write delay. MDTP achieves 13.54% improvement over Aria2 and 2.6% over Static Chunking under 0.5s latency. (c) Transfer time delta with and without 0.5s added latency, including disk write delay. MDTP and Aria2 improve by 0.52s and 1s under 0.5s latency, respectively while Static Chunking experiences an extra 6.24s latency. (d) Transfer time delta with and without 0.5s added latency, excluding disk write delay. MDTP and Aria2 had similar latency under 0.5s, while Static Chunking experienced nearly three times higher latency.
• Figure 4: (a) MDTP vs. Aria2 transfer time for 32GB and 64GB file sizes with and without bandwidth throttling. Bandwidth throttling increases transfer time for both MDTP and Aria2, with a greater impact on Aria2. (b) Transfer time delta of MDTP and Aria2 under different bandwidth conditions. The delta between MDTP and Aria2 increases with bandwidth throttling, as Aria2 does not utilize the slower replicas.
• Figure 5: (a) Percentage of available replica utilization for different file Sizes. MDTP and Static Chunking utilize all replicas, while Aria2 uses only 83% of them. (b) Number of packets sent to replicas for a 32GB file transfer. It shows MDTP and Static Chunking are balanced but Aria2 burdens the fastest replica. (c) Number of requests and request sizes in MDTP and Static Chunking for a 32GB File. MDTP demonstrates better load balancing compared to Static Chunking.