In this paper, we study the problem of designing experiments that are conducted on a set of units such as users or groups of users in an online marketplace, for multiple time periods such as weeks or months. These experiments are particularly useful to study the treatments that have causal effects on both current and future outcomes (instantaneous and lagged effects). The design problem involves selecting a treatment time for each unit, before or during the experiment, in order to most precisely estimate the instantaneous and lagged effects, post experimentation. This optimization of the treatment decisions can directly minimize the opportunity cost of the experiment by reducing its sample size requirement. The optimization is an NP-hard integer program for which we provide a near-optimal solution, when the design decisions are performed all at the beginning (fixed-sample-size designs). Next, we study sequential experiments that allow adaptive decisions during the experiments, and also potentially early stop the experiments, further reducing their cost. However, the sequential nature of these experiments complicates both the design phase and the estimation phase. We propose a new algorithm, PGAE, that addresses these challenges by adaptively making treatment decisions, estimating the treatment effects, and drawing valid post-experimentation inference. PGAE combines ideas from Bayesian statistics, dynamic programming, and sample splitting. Using synthetic experiments on real data sets from multiple domains, we demonstrate that our proposed solutions for fixed-sample-size and sequential experiments reduce the opportunity cost of the experiments by over 50% and 70%, respectively, compared to benchmarks.
翻译:在本文中,我们研究设计一系列单位,如在线市场用户或用户群体等用户或用户群体在数周或数月等多个时间段进行的实验的问题。这些实验对于研究对当前和未来结果产生因果关系的处理方法(即时和滞后效应)特别有用。设计问题涉及在试验之前或试验期间为每个单位选择一个治疗时间,以便最准确地估计瞬时和滞后效应、后期试验。这种优化治疗决定可以通过减少其抽样规模的要求,直接减少试验的机会基准成本。优化是一个NP硬的整数程序,我们为此提供了近乎最佳的解决方案,在设计决定全部在开始时(固定的缩略图设计)时,这些处理方法对当前和未来的结果都具有因果关系。接下来,我们研究连续的实验,允许每个单位在试验期间作出适应性决定,并有可能尽早停止实验,进一步降低其成本。然而,这些实验的顺序性质使得设计阶段和估计阶段都变得复杂。我们建议一种新的算法,即PGAE,通过适应性化的处理方式,来应对这些挑战,从我们分别对定期的处理决定进行比较,评估50个阶段,从实际的实验中,从利用模拟的模型模型分析,从我们用真实的模型分析中,到模拟的模型的模型中分别地分析,从模拟的模型学中,到绘制的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型的模型,到绘制。