Randomised field experiments, such as A/B testing, have long been the gold standard for evaluating software changes. In the automotive domain, running randomised field experiments is not always desired, possible, or even ethical. In the face of such limitations, we develop a framework BOAT (Bayesian causal modelling for ObvservAtional Testing), utilising observational studies in combination with Bayesian causal inference, in order to understand real-world impacts from complex automotive software updates and help software development organisations arrive at causal conclusions. In this study, we present three causal inference models in the Bayesian framework and their corresponding cases to address three commonly experienced challenges of software evaluation in the automotive domain. We develop the BOAT framework with our industry collaborator, and demonstrate the potential of causal inference by conducting empirical studies on a large fleet of vehicles. Moreover, we relate the causal assumption theories to their implications in practise, aiming to provide a comprehensive guide on how to apply the causal models in automotive software engineering. We apply Bayesian propensity score matching for producing balanced control and treatment groups when we do not have access to the entire user base, Bayesian regression discontinuity design for identifying covariate dependent treatment assignments and the local treatment effect, and Bayesian difference-in-differences for causal inference of treatment effect overtime and implicitly control unobserved confounding factors. Each one of the demonstrative case has its grounds in practise, and is a scenario experienced when randomisation is not feasible. With the BOAT framework, we enable online software evaluation in the automotive domain without the need of a fully randomised experiment.
翻译:A/B测试等随机实地实验长期以来一直是评估软件变化的金本位标准。在汽车领域,运行随机现场实验并非总是理想的、可能的、甚至是道德的。面对这些限制,我们开发了一个框架BOAT(Bayesian因果模型用于ObvservAtional测试),与Bayesian因果推断相结合,利用观测研究来理解复杂的汽车软件更新和帮助软件开发组织得出因果性结论对真实世界的影响。在这项研究中,我们在Bayesian框架中展示了三种因果推断模型及其相应的案例,以应对汽车领域通常经历的三种软件评估挑战。我们与我们的行业协作者一起开发了BOAT框架,通过对大批车辆进行实证性研究来展示因果关系。此外,我们将因果假设理论与其实际影响联系起来,目的是为如何在汽车软件工程中应用因果模型。我们在Bayesian框架及其相应案例中,当我们没有在不因果性操作性地进行平衡控制和处理时,当我们无法进行在线交易时,当我们无法在每次用户基础设计、测试时,需要精确地分析时,对机尾定的机底分析时,对机因反应的机因后果分析结果理论影响时,我们可以完全地分析,我们进行一种对结果的实验性处理,我们进行不需地分析,需要对结果分析。