Three types of tests

The old saying goes something like this:

There are three types of people in the world, those that agree with me, those that disagree with me, and those that will see the error of their ways (and switch camps.)

Of course, the moral of that often misquoted koan is that arbitrary groupings are almost as tedious as their mathematical underpinnings in set theory.  So, for the sakes of the insomniacs reading my One Shore blog, I offer the following sedative meta-semantic rambling:

There are three types of tests:  validation tests, regression tests, and exploratory tests.

What follows is my definitions of the three and some weak arguments for my particular arbitrary grouping.

Validation testing ensures that a product meets the requirements set forth in the design.  (Not all requirements are explicit — nor are all designs complete.)  Functional testing frequently falls into this category.  So too, does unit testing, which is really code level functional testing (or not actually testing at all — see my other posts on unit testing.)

Regression testing verifies that nothing has broken (or re-broken) when changes are made.  Smoke tests fall into this category, but can also include (full or partial) functional (manual or automated)  regression testing.  Regression tests can either verify that functional previous validated hasn’t broken or that bugs are not re-introduced (“have not regressed“).

Exploratory testing The idea is to find out what you can do, what you can’t do, and what you can do that you shouldn’t be able to.  You’re typical ad-hoc testing is exploratory.  Exploratory doesn’t necessarily mean ad-hoc.  It can be targeted or methodical.  Usability testing is often exploratory in nature.  Security and stability are best ensured through exploratory testing.   The key is that you’re not testing that the requirements have been met, or that known defects have not regressed…you’re exploring.

These groupings are more about what your goal is than how you test.  You can see some bleed over and contradiction in my trying to pidgeonhole traditional test groupings into my newer (and correcter) categories.

All three types of tests are important, and the there should be a balance of all three in any project.  I don’t want to say they’re equally important, because the emphasis depends on the project.

Some projects might have very specify requirements that it is imperative are met.  Validation would thus carry more weight.   An existing code base may require thorough regression after even a small change.  Exploratory testing allows for intuition, inference, and even chance to play a factor in testing.

Exploratory testing finds the most bugs, hands down.  And it also generated the most feature requests.  Which is maybe why it’s not often a favorite of management. The simple reason is that you cannot fully define the full scope inside a set of features or defects.   It allows the tester’s brain to “guess”, to infer requirements or intuit potential weak points.

Regression testing is preventative in nature.  A good set of automated regression tests are easily the best way maintain quality.  But exploratory tests and validation can be automated as well.  Random input generators are exploratory in nature.  Test first development is validation testing as regression testing.

And now even I need to take a nap.