The latest focus in the IT industry is Quality, Quality and Quality. Quality is required to ensure we do not run into problems such as the infamous dumps of windows, the core dumps of C, the out of memory of java and so on and so forth. Quality Assurance is a very important part of product development, no arguments here. But, I think it is very important to understand what the Quality Goals are, before embarking upon deciding the means of testing the product. I think everybody agrees having a hundred percent working product is a myth!!! In essence a Quality Goal basically defines what bugs I can live with and what bugs are show stoppers for my environment.
In my view a typical quality goal has definitely to be something in the lines of “Ensure that 95% of my customers do not run into problems when using my product” as opposed to in the lines of “all the code I have written has to be tested”. I think the later assumes “I have written code which a customer actually wants to use”, while the first say “possibly I have not written code that the customer wants, I want to recognize it”.
The problem of defining the goal as the later is that it directs the path taken for quality into different directions where effort and energy are wasted on unnecessary things such as unnecessary unit test cases and code coverage. Don’t get me wrong, I believe in “unit test cases”. But I believe in it a little differently from what is normally written in unit test cases. What I have seen mostly done for unit test cases is a test case is written for every class or set of classes coded. What is wrong in this!! Possibly the class or set of classes themselves are unnecessary. What is use of spending all this time in writing the code first and then the test cases for them? But, instead what really is required is “take the customer functionality defined” split them into “logical smaller pieces”, identify the tests for these smaller pieces and write them as unit test cases. Most of the time, writing test cases after coding and by the coder is pretty much useless, since the test cases are written usually for the code written and ensures nothing, if anything it ensures no one comes in and changes the existing code to work correctly, if it had originally been coded incorrectly.
As to “code coverage”, I have started wondering what code coverage really gives us when we consider our Quality Goal as defined above. Does this act as a way to measure quality of code, again I wonder, does not the test cases already do this? Does it act as a way to measure the quality of the test cases? Again, I don’t think so. I can very easily write a unit test case for a given class to call all the “code flow paths” and get a good code coverage percentage without actually testing the functionality. So, basically I have not measured the quality of the test cases but only the quantity of the test cases. So, I wonder what I get by measuring code coverage percent. If anything I believe what is more required is a way to detect “code missing” rather than how much code have test cases. When considered from regression perspective also I covering functional tests is more important than plain code coverage.
I think the main problem in today’s world of Quality assurance is that it works in 1 dimension only and that is of “code” while in actuality it has to be 2 dimensional that is of “design & code”.