The Test Design Pattern Template

Excerpted from Testing Object-Oriented Systems: Models, Patterns, and Tools. Copyright (C) 2000, Robert V. Binder, all rights reserved.

Patterns are General Solutions

A pattern is a general solution to a specific recurring design problem. It explains the insight and good practices that have evolved to solve the problem. It provides a concise definition of common elements, context, and essential requirements for a solution. Patterns were first used to describe constellations of structural elements in buildings and towns. For example, the "half-hidden garden" pattern considers how to frame a view of a garden so that onlookers have a sense of well-being and positive anticipation [Alexander 79].

Some designers of object-oriented systems began to express certain stereotypical relationships and class semantics following the style of architectural patterns [Coplien+95]. These software design patterns capture the essence of solutions to common software design problems. For example, the Model-View-Controller pattern suggests that three kinds of components should be used to provide an interactive display of stored values: classes that contain the stored information (the Model), classes for the user interface (the View), and classes that mediate interaction between the user interface and the content (the Controller) [Gamma+95]. Brad Appleton's Patterns in a Nutshell is an excellent introduction to design patterns.

The subjects, organization, and presentation of testing strategies in [Beizer 95] can be considered a pattern template for test design, suggesting the sections that a test design pattern should address. Each of the six chapters on generic testing strategies follows the same outline: (1) Relations and the Model, (2) Technique, (3) Application Indicators, (4) Bug Assumptions, (5) Limitations and Caveats, and (6) Automation and Tools.⁽¹⁾ Beizer's outline covers the essential questions that any test design approach must address. Each chapter is a pattern for test design. Siegel's "Test Models" for object-oriented development are similar in that the same subject outline is used for many testing problems: (1) Overview, (2) Leadership (general conditions and goals), (3) Structure (a model of the things to test ), (4) Dynamics (how to generate things that the model tests), (5) Example, and (6) Road Map (a graphic summary of the model) [Siegel 96]. Siegel's test models are less elaborate than Beizer's outline, but can also be considered as a kind of test design pattern.

The test design pattern template incorporates these test-specific elements and sections considered essential for any pattern template. The sections of the test design template follow. They are questions to the pattern writer, whose answers should articulate the insights, assumptions, and mechanics of the test strategy presented in the pattern.

Test Design Pattern Template

Name

A word or phrase that identifies the pattern and suggests its general approach.

Intent

What test design problem does this pattern solve? What is the test strategy? This is a very brief synopsis.

Context

In what circumstances does this pattern apply? To what kind of software entities? At what scope(s)? This section corresponds to the first problem to be solved in test design: given some implementation, what is an effective test design and execution strategy? This section corresponds to the "motivation," "forces," and "applicability" subjects of design patterns.

Fault Model

Why does this pattern work? What kind of bugs does this pattern target? What kinds of bugs are guaranteed to be found if they exist? What kinds of bugs can hide? This section explains how the test model meets the necessary conditions⁽²⁾ for revealing the targeted faults. For a test case to reveal a fault, three things must happen.

• The fault must be reached. The test input and state of the system under test must cause the code segment in which the fault is located to be executed.
• The failure must be triggered. A fault does not always produce incorrect results when it is reached. The test input and state of the system under test must cause the code segment in which the fault is located to produce incorrect result.
• The failure must be propagated. The incorrect result must become observable to the tester or an automated comparator.

This section must explain how the test suite will reach the targeted faults, how it can cause a failure to be triggered, and how a failure will be propagated to become observable.

Strategy

This section explains how the test suite is produced and implemented. There are four mandatory subsections.

• Test Model defines a representation for the responsibilities and/or implementation which are the focus of test design.
• Test Procedure defines an algorithm, technique, or heuristic by which test cases are produced from the model.
• Oracle defines the algorithm, technique, or heuristic by which actual results to a test case are to be evaluated for pass/no pass.
• Automation discusses automated approaches to test suite generation, test run execution, and test run evaluation. The appropriate extent of manual testing is discussed. In many cases, this section suggests which test automation design patterns are applicable.

The strategy is typically presented by example. The concrete result of applying a test design pattern is an uninteresting, possibly cryptic collection of input/output vectors. Instead of a commentary on test vectors, this section contains a step-by-step application of the test model to suitable example and the resulting test suite.

Entry Criteria

This section lists preconditions for effective and efficient testing with this pattern. An implementation should not be tested before it is, as a whole, test-ready.⁽³⁾ This has three beneficial effects. First, this reduces time lost because the IUT is too buggy to test. Second, this reduces waste of testing resources on bugs which can be more easily revealed and removed by an antecedent process, typically testing at a smaller scope. Third, this reduces the number of bugs that will escape from a given scope of testing. As test scope increases, the extent to which individual components can be exercised decreases. For example, when testing at use case scope, we are typically do not attempt to achieve statement coverage on each component that supports the use case. Instead, we try to achieve coverage of all the inter-component message paths in the use case. We cannot hope to achieve individually adequate coverage when we are testing at higher scope. If lower scope testing is inadequate or skipped, it is likely that many bugs will not be revealed by higher scope testing, even if it is comprehensive. Therefore, meeting the entry criteria will improve both efficiency and effectiveness of testing.⁽⁴⁾

Exit Criteria

What objective criteria must be met to achieve a complete test with this pattern? This section defines the extent to which the modeled responsibilities should be exercised and a corresponding coverage metric for an implementation at the scope (e.g., branch coverage at method scope.) or other conditions that indicate adequate testing has been achieved.

Consequences

What are the general prerequisites, costs, benefits, risks, and considerations for using this pattern?

Known Uses

What are the known uses of this test design pattern? What are the known uses of test models and strategies incorporated in this pattern? What are the efficiency and effectiveness of this pattern or similar strategies, as established by empirical studies?

Related Patterns

Are there any test design patterns that are similar or complementary? This section is omitted if there no related patterns.

Using Test Patterns

Test design patterns are compatible with most test processes. There are no essential differences in the basic tasks of testing when one uses a test pattern. Test patterns simply frame and focus the issues that must be resolved to accomplish any kind of testing. Patterns support results-oriented testing: that is, a scope-appropriate responsibility model of the implementation under test is needed. It is assumed that the IUT runs well enough to allow testing to proceed. The basic steps for using a test pattern are as follows.

1. Early in the development process, select test patterns that correspond to the scope and structure of the system under development and its parts.

2. Develop the test model for implementation under test.

3. Orchestrate your development and testing work so that the test suite will be applied to an implementation that meets the minimum operability threshold, as suggested in the entry criteria.

4. Generate the test suite by applying the test procedure to the test model.

5. Develop the test automation implementation.

6. Run and evaluate the tests. If the recommended coverage is not achieved, revise the test suite as needed.

Notes

1. The six design chapters are Control-Flow Testing, Loop Testing, Data-Flow Testing, Transaction-Flow Testing, Domain Testing, Syntax Testing, and Finite-State Testing.

2. These conditions are called the "ideal fault conditions" in [Marick 95] and the "fault/failure model" in [Friedman+95].

3. Test-ready was coined by Bob Poston. As an adjective, it means that test design or test execution may proceed without interference caused by missing or incomplete specifications and/or implementations. A test readiness assessment is a formal process activity defined in the SEI Capability Maturing Model which determines the test-readiness of a work-product [SEI 93].

4. Studies indicate that testing effectiveness reaches a plateau related to test scope [Horgan 96].

References

[Alexander 79] Christopher Alexander. The timeless way of building. New York: Oxford University Press, 1979.

[Beizer 95] Boris Beizer. Black box testing. New York: John Wiley & Sons, 1995.

[Buschmann+96] Frank Buschmann, Regine Meunier, Hans Rohnert, Peter Sommerlad, and Michael Stal. Pattern-Oriented Software Architecture: A System of Patterns. New York: John Wiley & Sons, 1996.

[Coplien+95] James Coplien and Douglas Schmidt (eds.) Pattern languages of program design. Reading, Mass.: Addison-Wesley, 1995.

[Firesmith 96] Donald G. Firesmith. Pattern language for testing object-oriented software. Object Magazine 5(9):32-28, January 1996.

[Friedman+95] Michael A. Friedman and Jeffery M. Voas. Software assessment: reliability, safety, testability. New York: John Wiley & Sons, Inc, 1995.

[Gamma+ 95] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design patterns: elements of reusable object-oriented software. Reading Mass.: Addison-Wesley, 1995.

[Horgan+96] Joseph R. Horgan and Aditya P. Mathur. Software testing and reliability. In Michael R. Lyu (ed). Handbook of software reliability engineering. Los Alamitos, Calif.: IEEE Computer Society Press, 1996.

[Marick 95] Brian Marick. The Craft of software testing. Englewood Cliffs, New Jersey: Prentice-Hall, Inc., 1995.

[SEI 93] Mark C. Paulk, Bill Curtis, Mary Beth Chrissis, and Charles V. Weber. Capability Maturity Model for Software, Version 1.1. CMU/SEI-93-TR-24. Pittsburgh: Software Engineering Institute, 1993.

[Siegel 96] Shel Siegel. Object-oriented software testing: a hierarchical approach. New York: John Wiley and Sons, 1996.