Scalable and Accurate Test Case Prioritization in Continuous Integration Contexts

Continuous Integration (CI) requires efficient regression testing to ensure the software systems' quality without delaying the CI builds significantly. This warrants the need for techniques to reduce the regression testing time, such as Test Case Prioritization (TCP) techniques that prioritize the execution of test cases such that faults can be detected as early as possible. Many recent TCP studies employ various Machine Learning (ML) techniques to deal with the dynamic and complex nature of CI. However, most of them use a limited number of features for training ML models and evaluate the models on subjects, for which the application of TCP is meaningless mainly due to the low regression testing time and the low number of failed builds. In this work, we first define, at a conceptual level, a data model that captures data sources and their relations in a typical CI environment. Second, we define a set of comprehensive features that account for the data model and cover all features previously used by the related studies. Third, we develop methods and tools to collect the defined features for 25 open-source software systems with enough failed builds whose regression testing takes at least five minutes. Forth, relying on the collected dataset containing a comprehensive feature set, we answer four research questions concerning the cost of data collection time, the accuracy of ML-based TCP based on the comprehensive feature set, the impact of the features on the accuracy, decay of ML-based TCP model over time, and the trade-off between the cost of the data collection time and accuracy of ML-based TCP techniques.