Software Quality Assurance: Testing Basics

Software is complex, in fact it can be infinitely complex depending on how the product is developed if extensibility is a factor. There are literally millions of scenarios that can occur not only the development of a product but even after a product has been deployed. Many times after a product is deployed (after getting customer feedback) developers will run into many issues that they did not foresee originally. When software is deployed and run on a client machine there are scenarios that may work for one user but not for another; like settings and runtime environments.

Because of these vast differences in environments, developers use common test utilities to cover their code and test against common scenarios that may break working code. Surely you should test to see if you code works, of course it should work you wrote it. There are, however, many scenarios that haven't been tested even after coverage is 100%. As such, after developing an application, there needs to be a standard for finding out possible failures in the code. I've noted some of the principles involved in testing basics for developing software.

• Find problems before customers do
• Find problems customers care about
• Ensure all features of the product are acceptable by the customer.
• Provide timely feedback on functionality of new features.
• Provide timely feedback on stability of old features.
• Role is not to prove the program works
• No way to fully test all possible program paths
• All software ships with some problems
• A problem is anything the customer doesn't like.

Some testing approaches have been known to include:

• White Box Testing (aka Glass Box Testing)
• Full access to the source code
• Static analysis
• Code reviews
• Static analysis tools (FxCop, etc)
• Dynamic Analysis
• Step through code in a debugger, examine program state
• Profiling (line coverage, branch coverage, performance)
• Black Box Testing
• No assumptions on implementation
• Two approaches
• Automate user interactions
• Use public API provided by the program
• Both techniques have value
• An UNDONE comment in code may reveal a subtle bug
• But customers don't read or profile code

Of course we can look into the types of tests here. Such as in Ad-Hoc tests, where the developers simply run the program and use it themselves. This makes it fairly easy and extremely quick to get coverage. Through ad-hoc tests we can get to major bugs very quickly. Next to Ad-Hoc is Manual tests, where developers write test cases to explicitly test the product while recording results and running them by hand. Opposed to both kinds is the Automated testing. In automated tests, a program is written to run the program automatically and test scenarios automatically. Generally automated test programs with the amount of code written can be much larger than the actual feature. In summation of the previous test types, is regression tests where developers run a test of test scenarios done in any other type of test.

Let's take a step back from testing and think about the psychology of testing itself.

1. Testing is inherently adversial
   
   • Programmers are trying to ship and you're telling them they can't.
   • Product managers are losing money in development costs and lost revenue and you're telling them they'll lose more if they ship now.
2. Never make it personal
   
   • Issues always outlast programmers, especially bad ones. When you write a test scenario, the code could be re-written and re-written over and over but the test scenario lives through all the code rewrites.
     
3. You have to carefully sell the things you really care about.
4. Clear expectations in bug reports
   
   • You can't simply put "X Doesn't Work." in a bug report. That doesn't explain anything to anyone reading this. A developer won't have your test code, you have to be able to explain enough to allow anyone to go right to where they need to be to fix code.

Aside from these basics, there are plenty of other scenarios that you will want to look into when in a proper test driven environment.