Practitioners and academics have long appreciated the benefits that experimentation brings to firms. For online web-facing firms, however, it still remains challenging in handling heterogeneity when experimental units arrive sequentially in online field experiments. In this paper, we study a novel online experimental design problem, which we refer to as the "Online Stratification Problem." In this problem, experimental units with heterogeneous covariate information arrive sequentially and must be immediately assigned into either the control or the treatment group, with an objective of minimizing the total discrepancy, which is defined as the minimum weight perfect matching between the two groups. To solve this problem, we propose a novel experimental design approach, which we refer to as the "Pigeonhole Design." The pigeonhole design first partitions the covariate space into smaller spaces, which we refer to as pigeonholes, and then, when the experimental units arrive at each pigeonhole, balances the number of control and treatment units for each pigeonhole. We analyze the theoretical performance of the pigeonhole design and show its effectiveness by comparing against two well-known benchmark designs: the match-pair design and the completely randomized design. We conduct extensive simulations to study the numerical performance of the different designs and conclude with practical suggestions.
翻译:长期以来, 执业者和学术界人士都认识到实验给公司带来的好处。 但是, 对于在线网上设计公司来说, 当实验单位相继抵达在线现场实验时, 处理不同性格的问题仍然具有挑战性。 在本文中, 我们研究一个新的在线实验设计问题, 我们称之为“ 在线分流问题 ” 。 在这个问题中, 实验单位具有各种千差万别的信息, 并且必须立刻被分配到控制或治疗组, 目标是将总差异最小化, 被定义为两个组之间最小体重的完美匹配。 为了解决这个问题, 我们提出一种新的实验设计方法, 我们称之为“ 皮眼设计 ” 。 鸽子洞设计首先将共变空间分割到较小的空间, 我们称之为“ 在线分流问题 ” 。 当实验单位到达每个信箱洞时, 平衡每个鸽子洞的控制和治疗单位的数量。 我们分析鸽子洞设计的理论性能, 并通过比较两个众所周知的基准设计来显示其有效性: 匹配式设计以及完全随机化的设计。 我们用大量的模拟来研究不同的数字设计。