Testing Concepts

Black-box testing and white-box testing are the two fundamental testing strategies. They are strategies, not technical or analytical methods.

Black-box tests are derived from the functional design specification, without regard to the internal program structure. Black-box testing tests the product against the end user, external specifications. Black-box testing is done without any internal knowledge of the product. It is important in practice to try to test, or at least write, detailed test plans for requirements and functional specifications tests without too much knowledge of the code. Understanding the code changes the way the requirements are seen and test design should not be “contaminated” by this knowledge too early.

Black-box testing will not test hidden functions (i.e., functions implemented but not described in the functional design specification), and the errors associated with them will not be found in black-box testing.

White-box tests require knowledge of the internal program structure and are derived from the internal design specification or the code. They will not detect missing functions (i.e., those described in the functional design specification but not supported by the internal specification or code).

BLACK-BOX METHODS FOR FUNCTION-BASED TESTS (page 83)

The following methods are commonly used:

· equivalence partioning

· boundary-value analysis

· error guessing.

The following are lesser-used methods:

· cause-effect graphing

· syntax testing

· state transition testing

· graph matrix.

Equivalence partitioning

Equivalence partitioning is a systematic process that identifies, on the basis of whatever information is available, a set of interesting classes of input conditions to be tested, where each class is representative of (or covers) a large set of other possible tests. If partitioning is applied to the product under test, the product is going to behave in much the same way for all members of the class.

The aim is to minimize the number of test cases required to cover these input conditions.

There are two distinct steps. The first is to identify the equivalence classes (ECs) and the second is to identify the test cases.

(1) Identifying equivalence classes

For each external input:

(i) If the input specifies a range of valid values, define one valid EC (within the range) and two invalid Ecs (one outside each end of the range).

Example: If the input requires a month in the range of 1-12, define one valid EC for months 1 through 12 and two invalid ECs (month<1>12).

(ii) If the input specifies the number (N) of valid values, define one valid EC and two invalid ECs (none, and more than N).

Example: If the input requires the titles of at least three but no more than eight books, then define one valid EC and two invalid ECs (<3>8 books).

(iii) If the input specifies a set of valid values, define one valid EC (within the set) and one invalid EC (outside the set).

Example: If the input requires one of the names TOM, DICK, or HARRY, then define one valid EC (using one of the valid names) and one invalid EC (using the name JOE).

(iv) If there is reason to believe that the program handles each valid input differently, then define one valid EC per valid input.

(v) If the input specifies a “must be” situation, define one valid EC and one invalid EC.

Example: If the first character of the input must be numeric, then define one valid EC where the first character is a number and one invalid EC where the first character is not a number.

(vi) If there is reason to believe that elements in an EC are not handled in an identical manner by the program, subdivide the EC into smaller ECs.

(2) Identifying test cases

(i) Assign a unique number to each EC.

(ii) Until all valid ECs have been covered by test cases, write a new test case covering as many of the uncovered ECs as possible.

(iii) Until all invalid Ecs have been covered by test cases, write a test case that covers one, and only one, of the uncovered invalid ECs.

(iv) If multiple invalid ECs are tested in the same test case, some of those tests may never be executed because the first test may mask other tests or terminate execution of the test case.

Equivalence partitioning significantly reduces the number of input conditions to be tested by identifying classes of conditions that are equivalent to many other conditions. It does not test combinations of input conditions.

Boundary-value analysis

Boundary-value analysis is a variant and refinement of equivalence partitioning, with two major differences:

First, rather than selecting any element in an equivalence class as being representative, elements are selected such that each edge of the EC is the subject of a test. Boundaries are always a good place to look for defects.

Second, rather than focusing exclusively on input conditions, output conditions are also explored by defining output ECs. What can be output? What are the classes of output? What should I create as an input to force a useful set of classes that represent the outputs that ought to be produced?

The guidelines for boundary-value analysis are:

· If an input specifies a range of valid values, write test cases for the ends of the range and invalid-input test cases for conditions just beyond the ends.

Example: If the input requires a real number in the range 0.0 to 90.0 degrees, then write test cases for 0.0, 90.0, -0.001, and 90.001.

· If an input specifies a number of valid values, write test cases for the minimum and maximum number of values and one beneath and beyond these values.

Example: If the input requires the titles of at least 3, but no more than 8, books, then write test cases for 2, 3, 8, and 9 books.

· Use the above guidelines for each output condition.

Boundary-value analysis is not as simple as it sounds, because boundary conditions may be subtle and difficult to identify. The method does not test combinations of input conditions.

Error guessing (page 87)

Error guessing is an ad hoc approach, based on intuition and experience, to identify tests that are considered likely to expose errors. The basic idea is to make a list of possible errors or error-prone situations and then develop tests based on the list. What are the most common error-prone situations we have seen before? Defects’ histories are useful. There is a high probability that defects that have been there in the past are the kind that are going to be there in the future.

Some items to try are:

· empty or null lists/strings

· zero instances/occurrences

· blanks or null characters in strings

· negative numbers

One of the studies done by Myers (1979) states that the probability of errors remaining in the program is proportional to the number of errors that have been found so far. This alone provides a rich source of focus for productive error guessing.

WHITE-BOX METHODS FOR INTERNALS-BASED TESTS

Once white-box testing is started, there are a number of techniques to ensure the internal parts of the system are being adequately tested and that there is sufficient logic coverage.

The execution of a given test case against program P will exercise (cover) certain parts of P’s internal logic. A measure of testedness for P is the degree of logic coverage produced by the collective set of test cases for P. White-box testing methods are used to increase logic coverage.

There are four basic forms of logic coverage:

(1) statement coverage

(2) decision (branch) coverage

(3) condition coverage

(4) path coverage.

White-box methods defined and compared

The following figure illustrates white-box methods. For example, to perform condition coverage, tests covering characteristics 1 and 3 are required. Tests covering 2 and 4 are not required. To perform multiple condition coverage, tests covering characteristics 1 and 4 are required. Such tests will automatically cover characteristics 1 and 2.

	Statement coverage	Decision coverage	Condition coverage	Decision/ Condition coverage	Multiple Condition coverage
1. Each statement is executed at least once	Y	Y	Y	Y	Y
2. Each decision takes on all possible outcomes at least once	N	Y	N	Y	implicit
3. Each condition in a decision takes on all possi-ble outcomes at least once	N	N	Y	Y	implicit
4. All possible combina-tions of condition out-comes in each decision occur at least once	N	N	N	N	Y

FIGURE: The white-box methods defined and compared. Each column in this figure represents a distinct method of white-box testing, and each row (1-4) defines a different test characteristic. For a given method (column), “Y” in a given row means that the test characteristic is required for the method. “N” signifies no requirement. “Implicit” means the test characteristic is achieved implicitly by other requirements of the method.

Exhaustive path coverage is generally impractical. However, there are practical methods, based on the other three basic forms, which provide increasing degrees of logic coverage.