Beyond ATE: Multi-Criteria Design for A/B Testing
Jiachun Li, Kaining Shi, David Simchi-Levi
TL;DR
This work addresses multi-objective adaptive experimentation with three core goals: minimizing cumulative welfare loss during the trial, achieving accurate estimation of heterogeneous treatment effects (CATE) in a non-parametric context, and preserving participant privacy under Joint Differential Privacy. The authors introduce ConSE, an instance-adaptive two-phase framework that partitions the covariate space via sequential dyadic segmentation to balance exploration and exploitation, achieving the Pareto-optimal frontier between regret and estimation error. They extend the framework with DP-ConSE, a privacy-preserving variant that incurs only negligible asymptotic cost in both regret and estimation accuracy while guaranteeing privacy for the final estimator. A key theoretical contribution is the tight, instance-dependent lower bound linking estimation error to regret through the instance complexity $H(\nu)$, along with mechanisms to robustly learn Pareto-optimal policies under indistinguishable instances. The results show that privacy can be incorporated without compromising post-experiment welfare, offering a principled approach for ethical, private, and efficient adaptive experiments in sensitive domains such as healthcare and high-stakes digital platforms.
Abstract
In the era of large-scale AI deployment and high-stakes clinical trials, adaptive experimentation faces a ``trilemma'' of conflicting objectives: minimizing cumulative regret (welfare loss during the experiment), maximizing the estimation accuracy of heterogeneous treatment effects (CATE), and ensuring differential privacy (DP) for participants. Existing literature typically optimizes these metrics in isolation or under restrictive parametric assumptions. In this work, we study the multi-objective design of adaptive experiments in a general non-parametric setting. First, we rigorously characterize the instance-dependent Pareto frontier between cumulative regret and estimation error, revealing the fundamental statistical limits of dual-objective optimization. We propose ConSE, a sequential segmentation and elimination algorithm that adaptively discretizes the covariate space to achieve the Pareto-optimal frontier. Second, we introduce DP-ConSE, a privacy-preserving extension that satisfies Joint Differential Privacy. We demonstrate that privacy comes ``for free'' in our framework, incurring only asymptotically negligible costs to regret and estimation accuracy. Finally, we establish a robust link between experimental design and long-term utility: we prove that any policy derived from our Pareto-optimal algorithms minimizes post-experiment simple regret, regardless of the specific exploration-exploitation trade-off chosen during the trial. Our results provide a theoretical foundation for designing ethical, private, and efficient adaptive experiments in sensitive domains.