Tech companies (e.g., Google or Facebook) often use randomized online experiments and/or A/B testing primarily based on the average treatment effects to compare their new product with an old one. However, it is also critically important to detect qualitative treatment effects such that the new one may significantly outperform the existing one only under some specific circumstances. The aim of this paper is to develop a powerful testing procedure to efficiently detect such qualitative treatment effects. We propose a scalable online updating algorithm to implement our test procedure. It has three novelties including adaptive randomization, sequential monitoring, and online updating with guaranteed type-I error control. We also thoroughly examine the theoretical properties of our testing procedure including the limiting distribution of test statistics and the justification of an efficient bootstrap method. Extensive empirical studies are conducted to examine the finite sample performance of our test procedure.
翻译:技术公司(如谷歌或脸书)经常使用随机在线实验和/或A/B测试,主要依据平均处理效果,将其新产品与旧产品进行比较,然而,同样至关重要的是,要发现质量处理效果,使新产品仅在某些特定情况下可能大大超过现有产品,本文件的目的是开发一个强大的测试程序,以便有效检测这种质量处理效果。我们提出了可扩展的在线更新算法,以实施测试程序。它有三个新颖之处,包括适应性随机化、顺序监测、以及以保障型号I的错误控制在线更新。我们还彻底审查我们的测试程序的理论性质,包括限制测试统计数据的分发和有效制靴方法的合理性。我们进行了广泛的实证研究,以审查测试程序的有限样本性能。