Test flakiness forms a major testing concern. Flaky tests manifest non-deterministic outcomes that cripple continuous integration and lead developers to investigate false alerts. Industrial reports indicate that on a large scale, the accrual of flaky tests breaks the trust in test suites and entails significant computational cost. To alleviate this, practitioners are constrained to identify flaky tests and investigate their impact. To shed light on such mitigation mechanisms, we interview 14 practitioners with the aim to identify (i) the sources of flakiness within the testing ecosystem, (ii) the impacts of flakiness, (iii) the measures adopted by practitioners when addressing flakiness, and (iv) the automation opportunities for these measures. Our analysis shows that, besides the tests and code, flakiness stems from interactions between the system components, the testing infrastructure, and external factors. We also highlight the impact of flakiness on testing practices and product quality and show that the adoption of guidelines together with a stable infrastructure are key measures in mitigating the problem.
翻译:试验的松散是一个主要的测试问题。 Flaky 测试显示非决定性结果,使连续整合受阻,并导致开发商调查虚假警报。工业报告显示,在大规模上,累积的闪烁性测试打破了对测试套件的信任,并带来巨大的计算成本。为了减轻这一影响,从业人员不得不查明闪烁性测试并调查其影响。为了阐明这种缓解机制,我们采访了14名从业人员,目的是查明(一) 测试生态系统中的不适应源,(二) 松散性的影响,(三) 从业人员在处理不适应性时采取的措施,以及(四) 这些措施的自动化机会。我们的分析表明,除了测试和编码外,系统各组成部分、测试基础设施和外部因素之间的相互作用也会产生不协调性。我们还强调了闪烁性对测试做法和产品质量的影响,并表明采用与稳定的基础设施是缓解问题的关键措施。