Some Thoughts On Software Testing and Software Test Engineering

Software testing is an artform, and, make no mistake about it, a good software test engineer is an artist.

Software testing often gets a bad rap as being “dull”, “boring” or something that the goofy wonk down the hall does, but that the hotshot developers don’t bother themselves with. I used to view testing as an evil necessity as well, until I discovered how challenging it could be.

Requirements – The Foundation for Testing

But, first off, I’d like to talk about requirements (which I seem to do a lot), since without requirements it is difficult, at best, to define meaningful tests. After the customer (be it an external customer or internal users, like, say, marketing) has outlined what they are looking for the functional requirements for the system can come into existence. The functional requirements should state specifically in clear and unambiguous language what the final product is supposed to do, but do not specifically state how it is supposed to do it. It is the functional requirements which will form the basis for the acceptance (or rejection) of the finished software product. Functional requirements might be in the form of well written use cases, or they could be something more formal. But regardless of the form, they must exist for any software product with any degree of complexity.

From the requirements one can derive the design, and from the design comes the code. Without the design there is no coherent plan for code implementation, and without the requirements there is no clear vision of what is to be built and what it is expected to do. These things all rely on one another and all need to be in place for a project to succeed and meet its initial objectives. The so-called “V-Model” helps to visualize the relationships between the requirements and design activities and the development of the various tests associated with each (see for more on the V-Model). The V-Model is itself a variation of the traditional waterfall, which has been largely superseded by variations of the iterative model. Nonetheless, it does illustrate the relationships between the requirements, design, code and test activities quite nicely. Please take a moment here and go look at it if you’ve not already seen it.

What Software Testing Is, and Isn’t

Now that we have a basic framework on which to hang the concepts of requirements, design and implementation, let’s talk about testing.

There seems to be a misconception about what, exactly, software testing is. One thing it is not is just testing the code after all the design and implementation is done. Functional acceptance testing is actually the least effective time to do a lot of testing, because if something has really gone off into the weeds the amount of effort necessary to pull it back in could well doom the project.

Another thing that software testing is not is simply exercising the code by examining the source and creating tests for it. That’s unit testing (described below) and while it is important, it is but one aspect of the whole testing and verification process.

Testing activities should start as soon as the first draft of the requirements are completed. This is when someone needs to start thinking about what kinds of tests will be needed to demonstrate that the requirements have been met. The project test plan and the initial draft of the functional test cases should come into existence along with the requirements.

Depending on how complex the project happens to be it may or may not need separate module and integration tests. If module and integration testing are deemed necessary then the test cases would be derived from the design, which, as mentioned earlier, is itself is derived from the requirements.

Lastly there are the unit tests. Unit tests utilize the smallest functional units of the software, typically individual functions or methods. Unit tests are often written by the person who wrote the code, although in some situations a separate person or group may be responsible for devising and executing the unit tests (this is the ideal situation, but the reality is more often that the resources to do this simply don’t exist).

The Challenges of Software Testing

The challenge for the software test case author then is to understand the project requirements and the software design. By becoming involved in the project from the outset the test engineer can help influence the design process to facilitate testing and help spot problems with the requirements or the design (or both). This is an important point, and one that is often overlooked.

For example, I’ve been involved in aerospace software projects where the requirements, detailed design documents and even the code were already in place before the test activities really started in earnest. These situations were usually a pain in the neck, with long hours spent pouring over the design documentation and reviewing the requirements and the requirements compliance matrix. It wasn’t uncommon for problems to crop up in the design, and sometimes even in the requirements. But by this time the requirements had already been “carved in stone” and getting them changed was a major effort. It would have been much easier if the test and verification activities had started a whole lot sooner.

The software test engineer needs to have a good grasp on the tools and techniques that will make the testing go more smoothly. Things like automated regression testing environments, MSC generators and logic modeling tools can be real time-savers in the long run. A software test engineer also needs to able to define and write a test harness for the code. A well-written test harness is invaluable, not only for the test activities but for the developers as well. The test harness provides a sandbox where new units and modules can be exercised with controllable inputs and their behavior easily observed and even logged for later review and analysis.

Creating the test cases, procedures, tools and environments for successful and effective software testing can be as challenging as creating the code itself, and sometimes even more so. If done with an eye towards the future much of the effort can be reused for future projects, thereby saving time and money over the long term.

References and Resources

There is a wealth of information on Wikipedia about software testing. There are also numerous books on the subject, some of which I’ve listed here:

Code Complete, 2nd Edition, Steve McConnell, Microsoft Press, 2004.

Software Testing, James D. McCaffrey, BookSurge Publishing, 2009.

Software Requirements Specification, Michael Jackson, ACM/Addison-Wesley, 1995.

Writing Effective Use Cases. Alistair Cockburn, Addison-Wesley, 2001.

Testing Embedded Software, Bart Broekman and Edwin Notenboom, Addison-Wesley Professional, 2002.

And, lastly, here are some useful resources on the Web:

Advanced Unit Testing by Marc Clifton

Effective Unit Testing

Dive Into Python (Chapter 13)

0 Responses to “Some Thoughts On Software Testing and Software Test Engineering”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow Crankycode on

Little Buddy

An awesome little friend

Jordi the Sheltie passed away in 2008 at the ripe old age of 14. He was the most awesome dog I've ever known.

%d bloggers like this: