Software Testing (or lack thereof)

“There is always a well-known solution to every human problem–neat, plausible, and wrong.” – H. L. Menken

So, what is software testing? The trite and obvious answer is that testing demonstrates that the software works correctly. But, what, exactly does that mean?

Does it mean that the compiler didn’t find any syntax errors? Or, does it mean that a unit or module generated an expected output for a particular input? In some cases it might mean that someone sat down and tried all the menus and buttons in a GUI according to a set of screenshots and didn’t notice any obvious errors (I’ve actually seen this one passed off as “complete functional testing”!).

The paradigm of software testing can include these things, but they are not the end of the story by any means. If nothing else, these activities do not speak to the real reason why one would want to do software testing in the first place, namely: Does the software do what it was intended to do and does it do it correctly, each time, every time? In order to know the answers with any degree of certainty one must go deep into the logic of the code and examine the dimly lit nooks and crannies for lurking bugs.

What the user sees in the interface, be it a fancy GUI or a simple set of lights in a cockpit, is but the tip of the iceberg. Just flipping a switch and observing that an indicator illuminates doesn’t really say much more than that the switch and the lamp are wired so that current flows. What logic senses the switch? What logic illuminates the lamp? What internal tests and actions are performed between the time the switch gets flipped and the lamp turns on? What happens if conditions are such that the lamp isn’t illuminated, or even worse, if it’s illuminated when it shouldn’t be?

A somewhat more formal definition of software testing might go like this: Software testing is a set of activities performed to insure that the software demonstrates that it meets all of its requirements, handles off-nominal conditions gracefully, and that all of the executable statements in the software are exercised by all possible inputs, both valid and invalid. As a corollary, an important point to always bear in mind was stated succinctly in the famous quote from Edsgar Dykstra: “Testing cannot demonstrate the absence of errors, only their presence.” This puts the onus on the testing to be as thorough as possible, lest a subtle defect go undetected until the user loses a day’s worth of work, or the train doesn’t stop when commanded, or the airliner falls out of the sky.

Software testing is not something one does alone in a dark room with no one looking. It is an integral part of the software life cycle, and deserves every bit as much attention as the requirements, design and coding phases. Unfortunately, in a typical high-pressure marketing-driven software shop things start to drop out of the life-cycle long before the product is delivered, starting with the requirements and design and then the testing, until all that is left is the coding.

Just as there are different levels of requirements (product description, functional requirements, implementation requirements and performance requirements, for example) there are also different levels of testing. Unfortunately I don’t think testing is as well understood as it should be (I also think that requirements are even less well understood, but that’s a different issue). For example, I have noticed over the course of my career that people often confuse unit testing for functional testing, and uttering the phrase “requirements coverage” amongst a typical group of software developers is almost certain to result in more than a few blank stares.

Rather than carry on about something that gets my bloodpressue elevated to begin with, I will instead take the easy way out and refer the interested reader to Wikipedia. Just type in “unit testing”, read what shows up and then follow the links.

Advertisements

0 Responses to “Software Testing (or lack thereof)”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s





%d bloggers like this: