A/B testing, or controlled experiments, is the gold standard approach to causally compare the performance of algorithms on online platforms. However, conventional Bernoulli randomization in A/B testing faces many challenges such as spillover and carryover effects. Our study focuses on another challenge, especially for A/B testing on two-sided platforms -- budget constraints. Buyers on two-sided platforms often have limited budgets, where the conventional A/B testing may be infeasible to be applied, partly because two variants of allocation algorithms may conflict and lead some buyers to exceed their budgets if they are implemented simultaneously. We develop a model to describe two-sided platforms where buyers have limited budgets. We then provide an optimal experimental design that guarantees small bias and minimum variance. Bias is lower when there is more budget and a higher supply-demand rate. We test our experimental design on both synthetic data and real-world data, which verifies the theoretical results and shows our advantage compared to Bernoulli randomization.
翻译:A/B测试,或受控实验,是因果比较在线平台算法性能的黄金标准方法。然而,在A/B测试中,传统的伯努利随机化在A/B测试中面临许多挑战,例如外溢效应和结转效应。我们的研究侧重于另一个挑战,特别是双面平台A/B测试 -- -- 预算限制。双面平台的买主往往预算有限,常规A/B测试可能无法应用,部分原因是分配算法有两种变式可能会相互冲突,如果同时实施,导致一些买主超出预算。我们开发了描述买方预算有限的双面平台的模式。我们随后提供了最佳的实验设计,保证了小的偏差和最小的差异。如果预算更多,供应需求率更高,则比前者低。我们用合成数据和真实世界数据测试我们的实验设计,这些数据可以核实理论结果,并显示我们比Bernoulli随机化的优势。