Usage-based Testing

Usage-based testing is one of the quality assurance components provided by EventBench. The rational behind usage-based testing is based on the fact that from a user's point of view, not all parts of a software are equally important. Therefore, from a user's point of view, the testing of all parts of software is not equally important, either. In usage-based testing, the testing effort is focused to reflect this difference in importance.

As a measure for the importance of a piece of software, we use the number of times a functionality of a software is used. This is based on observation of the software in form of usage sequences. A usage sequence is an ordered list of events. Based on the known data about a software, i.e., the observed usage sequences, stochastic processes are trained to define probabilistic usage profiles. The usage-based testing technique are based on the usage profiles.

Usage Profile Types

Currently, EventBench supports usage profiles based on for different stochastic proccesses:

• First-order Markov Model (FOMM): In FOMMs the probability of the next event only depends on the last observed events. Using a GUI application as example, this means that the next click of a user only depends on the last click. All previous click do not impact the probability. The advantages of FOMMs are their low complexity and high flexibility. With flexibility, we mean that a lot of scenarios have a positive probability in the FOMM, because only the last event impacts the probability. The drawback is that such models are not well suited to even model rather simple preconditions. As an example, consider a form where you need to check two boxes: that you accept a privacy agreement and that you accept the terms and conditions of a product. If only one click is remembered, it is impossible to know if both fields have been checked.
• Higher-order Markov Model (HOMM): In HOMMs the probability of the next event only depends on the k observed events, where k is a positive integer known as the Markov order of the HOMM. Using a GUI application as example, this means that the next click of a user only depends on the last k clicks. All clicks that are at least k+1 clicks ago do not impact the probability. The advantages and drawbacks of the HOMMs are the opposite of the FOMMs. The complexity is exponential in k and can easily get out of hand and the HOMMs are less flexible because they remember k events. On the other hand, if k clicks are remembered, it is possible to model pre-conditions that require up to k events to model.
• Prediction by Partial Match Model (PPMM): PPMMs are a combination of Markov models with different Markov orders. In principle, a k_max order PPMM is a HOMM with Markov order k_max. However, the PPMMs have an opt-out probability escape, with which a lower Markov order is used. This is bounded by a minimal Markov order k_min for which we set the opt-out probability to zero. The PPMMs, therefore, combine the advantages and drawbacks of the FOMMs and HOMMs in terms of modelling of pre-conditions and flexibility.
• Random Deterministic Finite Automata (RDFA): The RDFAs are a subtype of the FOMMs. They structure is the same as a FOMM, the difference lies in the calculation of the probabilities. In an RDFA, all possibly following symbols have the same probability.

Test Suite Generation Algorithms

TODO

Usage-based Coverage Criteria

TODO