Automated Software Testing

Automated Testing Improves Software Quality

The computer can do more tests of a program than we can manually and do them more reliably. In rigorous testing environments, such as software for jet aircraft, it is standard practice to measure the percentage of program statements being executed by the tests and aim for 100 percent coverage.

One simple way to start automated testing is to borrow the concept of the Pascal validation suite: a series of batch test jobs that tell whether a Pascal compiler conforms to the standard. A typical job tests one function, often by modifying data two different ways and comparing the results.

At Robelle, a test of our Suprtool product might copy a file, then copy it again with a system tool. The resulting files are compared. If there are any difference, the job aborts and the test fails. If there is no difference, the job runs to completion and deletes its $stdlist file. When Suprtool is revised, the test suite is run that night. It is amazing how often a seemingly minor change causes 10 or 20 test jobs to fail.

When working on a bug, a good practice is to add a test that reproduces the problem. When the bug is fixed, the test passes. Reproducing a bug in the test suite also provides a warning if the bug creeps back into the code. It is embarrassing how often old bugs resurface.

More ideas on software quality and productivity.