I have been talking to many people these days about code coverage, unit testing, and the ultimate question of “how many tests are enough?”. for those of us not up on all the buzzwords here is a quick TL;DR version of the buzzwords.

  • code coverage - how much of the application/library code is covered by unit testing.
  • unit testing - this is a set of tests that test the various functions/methods/subs of your application to make sure that they are operating like they are expected to.
  • QA - quality assurance. this is the process to make sure the application and its code is working like expected and there is no “bugs” (or as few as possible).
  • testing framework - every language has a testing framework that runs the applications unit tests. this framework not only runs the tests but is the base framework to make the tests from.
    • XojoUnit - Xojo’s unit testing framework.
    • TAP/Test::* - Perl’s unit testing framework.
    • PHPunit - PHP’s unit testing framework.
    • QUnit/JSUnitTesting - javascript’s unit testing (they are two competing frameworks).
    • and the list go on and on…

now back to the question(s), how much coverage and how many tests do we need? the short answer is code coverage should be 100% and enough tests to prove (to yourself) that there are no bugs. no reality is that we normally cant do 100% code coverage. 100% means every line of code we write is covered by one+ test(s). We can get most lines of code but I have seen very few places that every line of code is covered. I think the goal of 100% is great. shoot for the stars, land on the moon.

now for the question of how many tests? i set a goal of at a minimum of ten(10) tests per method plus at least one test for every bug found. i also try to add more when i can come up reasonable tests. i dont add more tests that dont help the goal of testing the code to make sure it works like it should.