BigFoot is a multi-platform client/server system implemented in C++. Clients use Microsoft Foundation Classes for Windows 95 and Windows NT. Server objects wrap a relational DBMS running on multiple Sun/Solaris servers. An ORB provides client/server communication using a distributed object model. BigFoot processing modes include high volume batch, real-time external links, and GUI-based interactive sessions. The system must provide 7x365 availability. Any incorrect output could have very serious and costly consequences.
The investment in test automation was motivated by short development cycles as well as the necessarily high reliability. BigFoot customers frequently produced totally new requirements on very short notice for rapid delivery. To obtain a repeatable reduction in development cycle time, BigFoot relied on systematic reuse and an architecture designed for extendibility . To insure high reliability and contribute to rapid development, the testing strategy had provide both high-volume regression testing and stringent component testing in compressed time cycles. Testing had to be both highly automated and highly effective. There were three main design goals for Tootsie.
(1) Effective user interface/ high usability. The target environment consisted of many technological islands. An effective test tool suite should present consistent mechanism for test documentation and execution, regardless of the technology of the implementation under test. Access to all test capabilities should be only a few "clicks" away from code editing, in any platform. We wanted Tootsie's users to:
- Control all test execution through a single user interface.
- Selectively run and rerun any subset of test cases in an entire test suite.
- Use integrated total-environment configuration management so that any application test package build could be run on any current and prior virtual machine configuration.
- Provide developers (individuals or small teams) an isolated "sandbox", to minimize interference, code dependencies, and inadvertent wipe-outs.
- Support any build cycle, from daily to monthly.
(2) Automate test generation. Hand-coding of drivers and stubs to for comprehensive and repeatable testing can easily double programming and maintenance effort. Some aspects of test design are very challenging, others are mechanical enough to be automated. Thus automated generation of test drivers, test cases, and stubs which would allow subsequent manual editing was needed. Tootsie would:
- Automatically generate tests from specifications and requirements.
- Automatically test hard-to-control capabilities like exception handling
- Automatically generate driver/stub test harness for a class or cluster with control over the extent of stubbing or integration.
- Support addition of manually prepared tests.
- Support cut and paste cloning of tests.
- Support regression testing by automatic impact analysis to select and re-run tests for unchanged components.
- Automate and support development of all levels of the FREE test design methodology: intra-class, cluster, subsystem, and system [1] [2] [3] [4].
(3) Automate test setup and execution. Effective testing cannot be done without a repeatable and controllable test suite. Repeatable testing is especially important with object-oriented development since adequate testing under most forms of reuse requires rerunning tests of reused components. It can be difficult to set and determine the state of a system of objects, so automated test setup is necessary part of repeatable testing. Tootsie's test setup and execution would:
Tootsie meet all of these requirements.
- Control the state of the system under test: set it, reset, save it.
- Measure coverage of statements, branches, and message interfaces.
- Represent expected results for a test to support automatic pass/fail evaluation.
- Perform automatic pass/fail evaluation.
- Provide automatic capture of test metrics.
The stale specification problem is addressed in two ways when an OOA/D repository is the primary source for test suites in interactive/incremental development. First, iterative changes tend to be small and therefore easier to record. Second, when tests can be automatically generated, there is a strong motivation to keep the specification current, since this reduces the developer's test design and test coding work. Automatic test generation, a viable repository, and iterative specification-based development produce a sustainable synergy which leads to higher quality systems.
Tootsie supports the FREE testing methodology. FREE is specification-based and requires that cartoon-level Object Modeling Technique models be upgraded to test-ready content. For BigFoot, this upgrade consisted of:
- Line Item Requirements. A concise narrative statement of each capabilities to be provided, with traceability to classes and to test cases.
- Extended Use-cases. A use case model with additional items: the operational relation and full integration with system-level dynamic object-interaction diagrams. The operational relation is a kind of decision table which can support systematic generation of external test cases. [3]
- A FREE state model for each class with sequentially constrained behavior.
- Pre and post conditions for each member function.
- State invariants for each class, implemented as Boolean member functions. A state invariant is a Boolean expression that defines the set of values allowed in the state. For example, the state invariant for the overdrawn state in an Account object might have the invariant bool IsOverdrawn{balance<0}
Without these extensions, the typical OMT specification is an untestable cartoon. The information is useful for human interpretation but cannot support automated test case generation.
A test plan for a system release is made up of many test runs. A test run may include many test cases; a test case may be used in many test runs. A test case has three main phases: setup, execute, and evaluate. The mechanics of these steps is significantly different depending on the implementation and kind of test. Even the simplest testing scenario involves using at least four testing tools, possibly running on different platforms. Without a common interface, the developer would have to learn to use ten completely different and idiosyncratic tool interfaces, ranging from command line cryptic to GUI obstructive. Tootsie was designed to present the same interface regardless of the test focus (class/cluster, subsystem, or application.) This was accomplished by using a multiplatform tool which could run any kind of test (Segue's QA/Planner and QA/Partner.) At the leaf level of a test plan, the appropriate test tool and implementation under test was designated. This provided a powerful yet straightforward interface to both large test plans (10000s of test cases) and the large collection of tools in Tootsie.
Table 1 lists the tools that comprised Tootsie. Readers should bear in mind that the tools used in Tootsie were selected to support the specific constraints and requirements of BigFoot.
| Slot | Tool |
|---|---|
| Line Item Requirements | DOORS |
| Extended Use Cases | DOORS and DXL |
| Object Models | Paradigm Plus |
| Test Case Generation | T/StP |
| Unit Test Driver Generation | Cantata |
| Code Coverage | Cantata |
| Regression Test Support | Discover |
| Test Suite Management | QA Organizer |
| GUI and Console Test Driver | QA Partner |
| Exception Test Simulation | QC/Sim |
| Configuration Management | ClearCase |
| Problem Tracking and Process Metrics | QA Plan/Track (Direct Technology) |
| Database Loader/Unloader | Custom Perl script and Sybase SQL |
| Database Comparator | Customer Perl script and Sybase SQL |
They're not necessarily the right choice for other applications. The challenge in developing Tootsie was in not so much in selecting the tools, but in providing bridges among these islands and the application islands. In a few cases, interfaces were available from the tool suppliers. For the majority we had to develop interfaces, which typically consisted of ASCII files and Perl scripts (not very exotic, but highly effective and portable.)
For example, scenario for unit test of a server class/cluster with Tootsie is comprised of the following steps (each is actually an activation of a separate test tool.) Each step is activated under the control of the QA/Partner.
- Extract the test parameters for the class from the OOA/D repository.
- Automatically generate the raw test case values.
- Generate a test harness (a driver, an object of the class under test, and a code section to instantiate a message for each test case) for the class.
- Include base class test drivers as appropriate.
- Generate a controllable stub library for untested objects used by the class under test.
- Instrument the class under test.
- Build a test package using the driver, instrumented class, and stub library.
- Run the database loader.
- Run the test package
- Run the database unloader
- Run the database comparator.
The output produced by this scenario includes recording of the test case results in the QA/Partner test script, the coverage report, the stub trace report (if any), the exception log, and the database comparator output. Tootsie provided a consistent interface and control mechanism for developers regardless of the scenario and scope of the test.
Tootsie supports a full life cycle test process using the FREE test design methodology. It is constructed from off-the-shelf testing tools and simple utility programs. Although this approach seemed straightforward to us, none of Tootsie's tool vendors could point to other organizations using the same extent of test automation. This correspondent would appreciate hearing from any readers who have developed similar testing systems -- drop me a note at rbinder@rbsc.com.
[1] Robert V. Binder, "State-based Testing," Object, v 5, n 6, July-August 1995.
[2] Robert V. Binder, "State-based Testing: Sneak paths and Conditional Transitions," Object, v 5, n 6, July-August 1995.
[3] Robert V. Binder, "Use-cases, Threads, and Relations: The FREE Approach to System Testing," Object, v 6, n 2, February 1996.
[4] Robert V. Binder, The FREE Approach to Object-Oriented Testing.