Adaptive Execution Scheduler for DataDios SmartDiff
Aryan Poduri
TL;DR
The paper tackles tail-latency and memory-safety in a single-differencing engine by treating differencing as a schedulable workload and introducing an adaptive execution scheduler that gates between in-memory and Dask backends while tuning batch size $b$ and workers $k$ under CPU and RAM budgets to minimize $p95$ latency. It presents a conservative working-set estimator, a memory-safe online cost/memory model, and a guarded hill-climb controller with backpressure, achieving low overhead integration with SmartDiff. The results show substantial improvements in $p95$ latency (roughly $23$–$40\%$ depending on baselines) and memory reductions (roughly $16$–$32\%$) with zero OOMs and stable throughput. This method enables safer, faster tail-latency performance for large-scale data migration and validation tasks without manual tuning.
Abstract
We present an adaptive scheduler for a single differencing engine (SmartDiff) with two execution modes: (i) in-memory threads and (ii) Dask based parallelism. The scheduler continuously tunes batch size and worker/thread count within fixed CPU and memory budgets to minimize p95 latency. A lightweight preflight profiler estimates bytes/row and I/O rate; an online cost/memory model prunes unsafe actions; and a guarded hill-climb policy favors lower latency with backpressure and straggler mitigation. Backend selection is gated by a conservative working-set estimate so that in-memory execution is chosen when safe, otherwise Dask is used. Across synthetic and public tabular benchmarks, the scheduler reduces p95 latency by 23 to 28 percent versus a tuned warm-up heuristic (and by 35 to 40 percent versus fixed grid baselines), while lowering peak memory by 16 to 22 percent (25 to 32 percent vs. fixed) with zero OOMs and comparable throughput.