Software testing is an artform, and, make no mistake about it, a good software test engineer is an artist.
Software testing often gets a bad rap as being “dull”, “boring” or something that the goofy wonk down the hall does, but that the hotshot developers don’t bother themselves with. I used to view testing as an evil necessity as well, until I discovered how challenging it could be.
Requirements – The Foundation for Testing
But, first off, I’d like to talk about requirements (which I seem to do a lot), since without requirements it is difficult, at best, to define meaningful tests. After the customer (be it an external customer or internal users, like, say, marketing) has outlined what they are looking for the functional requirements for the system can come into existence. The functional requirements should state specifically in clear and unambiguous language what the final product is supposed to do, but do not specifically state how it is supposed to do it. It is the functional requirements which will form the basis for the acceptance (or rejection) of the finished software product. Functional requirements might be in the form of well written use cases, or they could be something more formal. But regardless of the form, they must exist for any software product with any degree of complexity.
From the requirements one can derive the design, and from the design comes the code. Without the design there is no coherent plan for code implementation, and without the requirements there is no clear vision of what is to be built and what it is expected to do. These things all rely on one another and all need to be in place for a project to succeed and meet its initial objectives. The so-called “V-Model” helps to visualize the relationships between the requirements and design activities and the development of the various tests associated with each (see http://en.wikipedia.org/wiki/V-Model_(software_development) for more on the V-Model). The V-Model is itself a variation of the traditional waterfall, which has been largely superseded by variations of the iterative model. Nonetheless, it does illustrate the relationships between the requirements, design, code and test activities quite nicely. Please take a moment here and go look at it if you’ve not already seen it.
What Software Testing Is, and Isn’t
Now that we have a basic framework on which to hang the concepts of requirements, design and implementation, let’s talk about testing.
There seems to be a misconception about what, exactly, software testing is. One thing it is not is just testing the code after all the design and implementation is done. Functional acceptance testing is actually the least effective time to do a lot of testing, because if something has really gone off into the weeds the amount of effort necessary to pull it back in could well doom the project.
Another thing that software testing is not is simply exercising the code by examining the source and creating tests for it. That’s unit testing (described below) and while it is important, it is but one aspect of the whole testing and verification process.
Testing activities should start as soon as the first draft of the requirements are completed. This is when someone needs to start thinking about what kinds of tests will be needed to demonstrate that the requirements have been met. The project test plan and the initial draft of the functional test cases should come into existence along with the requirements.
Depending on how complex the project happens to be it may or may not need separate module and integration tests. If module and integration testing are deemed necessary then the test cases would be derived from the design, which, as mentioned earlier, is itself is derived from the requirements.
Lastly there are the unit tests. Unit tests utilize the smallest functional units of the software, typically individual functions or methods. Unit tests are often written by the person who wrote the code, although in some situations a separate person or group may be responsible for devising and executing the unit tests (this is the ideal situation, but the reality is more often that the resources to do this simply don’t exist).
The Challenges of Software Testing
The challenge for the software test case author then is to understand the project requirements and the software design. By becoming involved in the project from the outset the test engineer can help influence the design process to facilitate testing and help spot problems with the requirements or the design (or both). This is an important point, and one that is often overlooked.
For example, I’ve been involved in aerospace software projects where the requirements, detailed design documents and even the code were already in place before the test activities really started in earnest. These situations were usually a pain in the neck, with long hours spent pouring over the design documentation and reviewing the requirements and the requirements compliance matrix. It wasn’t uncommon for problems to crop up in the design, and sometimes even in the requirements. But by this time the requirements had already been “carved in stone” and getting them changed was a major effort. It would have been much easier if the test and verification activities had started a whole lot sooner.
The software test engineer needs to have a good grasp on the tools and techniques that will make the testing go more smoothly. Things like automated regression testing environments, MSC generators and logic modeling tools can be real time-savers in the long run. A software test engineer also needs to able to define and write a test harness for the code. A well-written test harness is invaluable, not only for the test activities but for the developers as well. The test harness provides a sandbox where new units and modules can be exercised with controllable inputs and their behavior easily observed and even logged for later review and analysis.
Creating the test cases, procedures, tools and environments for successful and effective software testing can be as challenging as creating the code itself, and sometimes even more so. If done with an eye towards the future much of the effort can be reused for future projects, thereby saving time and money over the long term.
References and Resources
There is a wealth of information on Wikipedia about software testing. There are also numerous books on the subject, some of which I’ve listed here:
Code Complete, 2nd Edition, Steve McConnell, Microsoft Press, 2004.
Software Testing, James D. McCaffrey, BookSurge Publishing, 2009.
Software Requirements Specification, Michael Jackson, ACM/Addison-Wesley, 1995.
Writing Effective Use Cases. Alistair Cockburn, Addison-Wesley, 2001.
Testing Embedded Software, Bart Broekman and Edwin Notenboom, Addison-Wesley Professional, 2002.
And, lastly, here are some useful resources on the Web:
Advanced Unit Testing by Marc Clifton
Effective Unit Testing
Dive Into Python (Chapter 13)