注射试验令依赖性 (On the Use of Mutation in Injecting Test Order-Dependency)

Background: Test flakiness is identified as a major issue that compromises the regression testing process of complex software systems. Flaky tests manifest non-deterministic behaviour, send confusing signals to developers, and break their trust in test suites. Both industrial reports and research studies highlighted the negative impact of flakiness on software quality and developers' productivity. While researchers strive to devise solutions that could help developers addressing test flakiness, the elaboration and assessment of these solutions are hindered by the lack of datasets large enough to leverage learning techniques. Aim: To address this lack, we conduct an exploratory study that investigates a new mean for producing datasets of flaky tests. Method: We propose an approach that relies on program mutation to inject flakiness in software tests. In particular, we plan to delete helper statements from tests to make their outcomes order-dependent, i.e., pass in certain running orders but fail in other orders. We intend to apply our mutation-based approach to a set of 14 Java projects to assess the effectiveness of test mutation in injecting order-dependency and generate a new dataset that could be used to study flakiness.

翻译：试验偏差被确定为影响复杂软件系统回归测试过程的一个主要问题。 Flaky 测试显示非决定性行为,向开发者发出混乱的信号,并打破他们对测试套件的信任。工业报告和研究都强调了不成熟对软件质量和开发者生产力的负面影响。研究人员努力制定解决方案,帮助开发者解决测试不成熟问题,但对这些解决方案的阐述和评估却因缺乏足够大的数据集而无法利用学习技术而受到阻碍。目标:为解决这一缺陷,我们开展了一项探索性研究,以调查产生片断测试数据集的新手段。方法:我们提议一种依靠程序突变的方法,以在软件测试中注入偏差性。特别是,我们计划从测试中删除帮助其结果的报表,以便使其结果依附于顺序,即某些运行订单通过,但其他订单失败。我们打算对一组14个爪哇项目采用基于突变法的方法,以评估注射顺序依赖性测试的有效性。方法:我们建议采用新的数据套件,以便用来研究休克状态。