Combinatorial Sequence Testing Using Behavioral Programming and Generalized Coverage Criteria

We present a new model-based approach to testing systems whose test vectors are sequences of actions and assertions (called events in this paper). Specifically, we propose an end-to-end solution to testing that includes a method for defining tests, a way to quantify testing quality, and a framework for assessing risks. We use behavioral programming to model what to test and present how its composable and incremental nature makes it effective for designing maintainable test plans. We then formalize a framework that generalizes previous approaches for specifying coverage criteria and show how to extract effective test suites from test plans according to these criteria. For readiness quantification, we present a Bayesian approach for measuring the probabilities of bugs, i.e., risk. We also demonstrate how this quantification can be used for an informed exploitation-exploration balance in testing. We present a proof-of-concept publicly available tool that demonstrates the completeness of the approach. Finally, we provide an empirical evaluation that demonstrates the effectiveness of our tool in finding bugs, assessing risks, and achieving coverage on real systems.