Given an unexpected change in the output metric of a large-scale system, it is important to answer why the change occurred: which inputs caused the change in metric? A key component of such an attribution question is estimating the counterfactual: the (hypothetical) change in the system metric due to a specified change in a single input. However, due to inherent stochasticity and complex interactions between parts of the system, it is difficult to model an output metric directly. We utilize the computational structure of a system to break up the modelling task into sub-parts, such that each sub-part corresponds to a more stable mechanism that can be modelled accurately over time. Using the system's structure also helps to view the metric as a computation over a structural causal model (SCM), thus providing a principled way to estimate counterfactuals. Specifically, we propose a method to estimate counterfactuals using time-series predictive models and construct an attribution score, CF-Shapley, that is consistent with desirable axioms for attributing an observed change in the output metric. Unlike past work on causal shapley values, our proposed method can attribute a single observed change in output (rather than a population-level effect) and thus provides more accurate attribution scores when evaluated on simulated datasets. As a real-world application, we analyze a query-ad matching system with the goal of attributing observed change in a metric for ad matching density. Attribution scores explain how query volume and ad demand from different query categories affect the ad matching density, leading to actionable insights and uncovering the role of external events (e.g., "Cheetah Day") in driving the matching density.
翻译:鉴于大规模系统产出指标发生意想不到的变化,必须回答为什么发生这种变化的原因:哪些投入导致度量的变化?这种归因问题的一个关键组成部分是估计反事实:由于一个输入的某个特定变化,系统指标的(假设性)变化;然而,由于系统各部分之间固有的随机性和复杂的相互作用,很难直接模拟一个产出指标。我们利用系统的计算结构将模拟任务分成分部分,这样每个分部分都对应一个更稳定的机制,可以精确地模拟需求的变化;这样一个归因问题的一个关键组成部分是估计反事实:由于一个输入的某个特定变化,系统参数的(假设性)变化;然而,由于系统各部分之间固有的随机性变化和复杂的相互作用,我们很难直接模拟一个产出指标。我们利用一个系统的计算结构结构结构,将每个分部分与一个更稳定的机制相对应,而每个分部分则可以精确地模拟一个更精确的变数, 与过去关于因因因果而变数的变数的计算结果相比,我们提出的方法可以将一个观察到的成型数据结果归为一个比我们所观察到的变数的变数。