Stability of test criteria and fault hierarchies in software testing
Software testing is an important activity to verify and validate a system. A
test criterion is a set of rules that are used for assessing the system under
test. The effectiveness of a test criterion is measured in terms of its ability to
reveal faults. Two key issues in software testing are: (a) effectiveness of test
criteria in detecting faults, and (b) minimisation of test effort. These issues
are studied empirically and formally within the framework of fault domain
and test hypothesis.
Typically, for a given test criterion, more than one test set may satisfy the
criterion for a specification and implementation. A new notion of stability
of test criteria is defined to assess the variation in effectiveness of test sets
for a given test criterion. Experimental evaluation of stability is performed
for various types of coverage such as condition coverage, decision condition
coverage, full-predicate coverage, modified condition decision coverage and
reinforced condition decision coverage.
Fault detection effectiveness is also studied using a formal framework,
which is applied to identify the conditions for the detection of various fault
classes. It is shown that the number of test cases in a test set that can detect
all hypothesised faults in an implementation depends on the complexity of the
One of the main difficulties with the fault-based testing approach is due to
the large number of possible elements in a fault domain or that can be generated
on the basis of a test hypothesis. To overcome this problem, various
conditions to establish fault hierarchies that help to identify stronger faults
are described. Here, the objective is to establish the relationship between
faults such that detection of one fault guarantees the detection of another.
The analysis of fault hierarchies is also shown to be useful in validating the
coupling hypothesis, which states that if a test technique can detect an implementation
with one fault, it can also detect the presence of multiple faults.
The results obtained from the empirical and formal analysis provide an
insight into earlier contradictory results regarding the effectiveness of test
criteria. The formal framework helps in the classification of specifications
and implementations in order to evaluate the effort required for testing; and
the concept of fault hierarchy is useful in reducing test effort.