In online experiments where the intervention is only exposed, or "triggered", for a small subset of the population, it is critical to use variance reduction techniques to estimate treatment effects with sufficient precision to inform business decisions. Trigger-dilute analysis is often used in these situations, and reduces the sampling variance of overall intent-to-treat (ITT) effects by an order of magnitude equal to the inverse of the triggering rate; for example, a triggering rate of $5\%$ corresponds to roughly a $20x$ reduction in variance. To apply trigger-dilute analysis, one needs to know experimental subjects' triggering counterfactual statuses, i.e., the counterfactual behavior of subjects under both treatment and control conditions. In this paper, we propose an unbiased ITT estimator with reduced variance applicable for experiments where the triggering counterfactual status is only observed in the treatment group. Our method is based on the efficiency augmentation idea of CUPED and draws upon identification frameworks from the principal stratification and instrumental variables literature. The unbiasedness of our estimation approach relies on a testable assumption that the augmentation term used for covariate adjustment equals zero in expectation. Unlike traditional covariate adjustment or principal score modeling approaches, our estimator can incorporate both pre-experiment and in-experiment observations. We demonstrate through a real-world experiment and simulations that our estimator can remain unbiased and achieve precision improvements as large as if triggering status were fully observed, and in some cases can even outperform trigger-dilute analysis.
翻译:在仅暴露干预的在线实验中,或“触发”的实验中,对于一小部分人口来说,使用减少差异技术来估计治疗效果,并足够精确地为商业决策提供信息,至关重要。在这些情况下,往往使用触发光极分析,以相当于触发率反比的幅度来减少总体意图对结果(ITT)影响的抽样差异;例如,5美元触发率相当于减少差异约20美元。应用触发光极分析,需要了解实验对象触发反事实状态的实验对象,即治疗和控制条件下主体的反事实行为。在本文中,我们建议使用一个不带偏见的ITT估计值,在实验中可减少差异,而触发事实状况只能由治疗组来观察;例如,5美元触发率的触发率相当于减少约20美元的差异减少20美元。为了应用触发光细微分析,需要了解实验对象触发反事实状态的实验对象,即,即在治疗和控制条件下,实验对象的反事实行为行为。 本文建议采用不偏差的ICTT值估计值计算方法,在模拟模型中,在进行实际变差的模型分析时,在进行实际变差分析时,我们用于进行原变变变变变变变的实验前的计算时可以完全显示。