In online experiments where the intervention is only exposed, or "triggered", for a small subset of the population, it is critical to use variance reduction techniques to estimate treatment effects with sufficient precision to inform business decisions. Trigger-dilute analysis is often used in these situations, and reduce the sampling variance of overall intent-to-treat (ITT) effects by an order of magnitude equal to the inverse of the triggering rate; for example, a triggering rate of $5\%$ corresponds to roughly a $20x$ reduction in variance. To apply trigger-dilute analysis, one needs to know experimental subjects' triggering counterfactual statuses, i.e., the counterfactual behavior of subjects under both treatment and control conditions. In this paper, we propose an unbiased ITT estimator with reduced variance applicable for experiments where the triggering counterfactual status is only observed in the treatment group. Our method is based on the efficiency augmentation idea of CUPED and draws upon identification frameworks from the principal stratification and instrumental variables literature. The unbiasedness of our estimation approach relies on a testable assumption that the augmentation term used for covariate adjustment equals zero in expectation. Unlike traditional covariate adjustment or principal score modeling approaches, our estimator can incorporate both pre-experiment and in-experiment observations. We demonstrate through both a real-world experiment and simulations that our estimator can remain unbiased and achieve precision improvements as large as if triggering status were fully observed, and in some cases can even outperform trigger-dilute analysis.
翻译:在仅暴露干预的在线实验中,或“触发”的实验中,对于一小部分人口来说,至关重要的是使用减少差异技术来估计治疗效果,并足够精确地为商业决策提供信息。在这些情况下,往往使用触发式稀释分析,并用相当于触发率反转的幅度级来减少总体意图-治疗(ITT)效应的抽样差异;例如,5美元触发率相当于减少差异约20美元。为了进行触发式缓解分析,需要了解实验对象触发反事实状态的精确度技术,即治疗和控制条件下各主体的反事实行为。在本文中,我们建议使用一个不带偏见的ITT估计值,在实验中可减少差异,而触发反事实状态只在治疗组中观察到。我们的方法以CUPED效率增强概念为基础,并借鉴本精度和工具变量文献中的识别框架。我们估算方法的公正性取决于一个可测试的假设,即:在治疗和控制条件下,在实验前,我们用于进行最大变异性调整时,在进行最大变异性分析时,我们用于进行最大变异性分析,在实验前,我们作为正变式试验前,我们可以继续展示。