Prototypical QA Site Case Studies

Here are three hypothetical organizations that could benefit from One Shore consulting.

I’ll try to write these up into the website:

The following are composite potential customers and not real organizations.

QUICKER RAMP UP TIME, QA PROCESS IMPROVEMENT

Startup X  is growing rapidly.   Originally, the founders did all the coding and testing (as well as marketing and janitorial work) themselves, but now are busy managing the company and (window) shopping for private jets.

With new hires comes different coding styles and varied skillsets, and the original authors barely know the code base anymore (when they do have time to look into it.)  The last release slipped due to some last minute bugs discovered, and ongoing feature creep.   They need to introduce some discipline into the development process and ensure quality is maintained and deadlines don’t slip.

They know they need a test environment and a better build & deployment process, but don’t have the time and resources to do it themselves.

Proposed solution: a managed virtual test environment from One Shore.

A test lab is set up within one week.  No hardware needs purchased, no firewalls need penetrated, or permissions granted.  With every checkin, the new code is built and deployed automatically.   Smoke tests then run against the test environment and problems are detected immediately.  Releases run smoothly, and manual testing is quicker too, since a developer or tester doesn’t need to redeploy the whole project (and populate sample data) every time a change is made.

OPEN SOURCE AUTOMATION TOOLS EQUALS SAVINGS

Corporation Y is a large enterprise.  They have a rigorous testing process and use expensive proprietary tools.  Their license is expiring, and rather than renewing it they want to investigate using open source tools.

Some of the team members are advocates of open source and agile, but don’t know how best to persuade management that it’s safe.  A pilot project is proposed, and open source tools identified.  While they know what they want and know their proprietary products well, they don’t have the experience with the open source equivalents and don’t know the limitations.

Proposed solution: open source automation tool training and consulting by One Shore.

A report detailing the features and limitations of comparable open source tools tools is presented.  A workshop and some pair-programming helps SDETs quickly see how new tests can be written using the open source framework.   Migration of legacy tests can be outsourced to One Shore and reviewed internally by testers whose main focus can be on the new features.

ON DEMAND DOMAIN EXPERTISE NEEDED (OCCASIONALLY)

Company Z is not a “tech” business.  However, they do have a small in-houce IT staff that does occasional updates to their custom software application.  Because it is such a specialized field, it took a lot of time to train their tester, and they were reluctant to let her go, but with sometimes several months between releases, they didn’t see any alternative.

They tried staffing agencies, but besides the training time, the search for a qualified tester took an inordinate amount of time as well.  What they really need is someone who can hit the ground running, know their product, provides reliable results, and is willing to work only one month in three, part time.

Proposed solution: part time staffing from One Shore.

While there will still be the initial training time, consulting is what we do. Having multiple clients allows us to give as much (0r as little) time as needed to a client, and have top notch staff willing to work part time on your project.  Because of the variety of experiences, best practices are assured, and because we’ve worked with you in the past, there’s no headhunting and retraining time waste.

Three types of tests

The old saying goes something like this:

There are three types of people in the world, those that agree with me, those that disagree with me, and those that will see the error of their ways (and switch camps.)

Of course, the moral of that often misquoted koan is that arbitrary groupings are almost as tedious as their mathematical underpinnings in set theory.  So, for the sakes of the insomniacs reading my One Shore blog, I offer the following sedative meta-semantic rambling:

There are three types of tests:  validation tests, regression tests, and exploratory tests.

What follows is my definitions of the three and some weak arguments for my particular arbitrary grouping.

Validation testing ensures that a product meets the requirements set forth in the design.  (Not all requirements are explicit — nor are all designs complete.)  Functional testing frequently falls into this category.  So too, does unit testing, which is really code level functional testing (or not actually testing at all — see my other posts on unit testing.)

Regression testing verifies that nothing has broken (or re-broken) when changes are made.  Smoke tests fall into this category, but can also include (full or partial) functional (manual or automated)  regression testing.  Regression tests can either verify that functional previous validated hasn’t broken or that bugs are not re-introduced (“have not regressed“).

Exploratory testing The idea is to find out what you can do, what you can’t do, and what you can do that you shouldn’t be able to.  You’re typical ad-hoc testing is exploratory.  Exploratory doesn’t necessarily mean ad-hoc.  It can be targeted or methodical.  Usability testing is often exploratory in nature.  Security and stability are best ensured through exploratory testing.   The key is that you’re not testing that the requirements have been met, or that known defects have not regressed…you’re exploring.

These groupings are more about what your goal is than how you test.  You can see some bleed over and contradiction in my trying to pidgeonhole traditional test groupings into my newer (and correcter) categories.

All three types of tests are important, and the there should be a balance of all three in any project.  I don’t want to say they’re equally important, because the emphasis depends on the project.

Some projects might have very specify requirements that it is imperative are met.  Validation would thus carry more weight.   An existing code base may require thorough regression after even a small change.  Exploratory testing allows for intuition, inference, and even chance to play a factor in testing.

Exploratory testing finds the most bugs, hands down.  And it also generated the most feature requests.  Which is maybe why it’s not often a favorite of management. The simple reason is that you cannot fully define the full scope inside a set of features or defects.   It allows the tester’s brain to “guess”, to infer requirements or intuit potential weak points.

Regression testing is preventative in nature.  A good set of automated regression tests are easily the best way maintain quality.  But exploratory tests and validation can be automated as well.  Random input generators are exploratory in nature.  Test first development is validation testing as regression testing.

And now even I need to take a nap.

QA Site competition (of a sort)

I noticed that JumpBox has hosted wikis (mediawiki, docuwiki, moinmoin), blogs (wordpress, movable type) project management (projectpier, trac, redmine), bug tracking (bugzilla, mantis), monitoring (nagios/cacti, zenoss) and version control (subversion) tools.

Which means that you could get a jumpbox (or combination of jumpboxes) and do what the core of a QA Site does. Which means, in one sense, that there’s competition, of a sort, but in another sense, that there’s a demand for these hosted applications.

They don’t have them all put together in one package, though they soon might, but they don’t have a dashboard and integration. That’s what a QA Site will provide, and it may very well be an open source wrapper project the way WAMP provides an Apache + Mysql + PHP development package for Windows. So I’m not too worried there.

Jumpbox may even have interest in hosting QA Site packages, with a version control, bug tracking, project management, documentation, (and possibly even continuous integration and test case management) tools. Or I might use JumpBoxes as a basis for QA Sites.

The value I’m proposing isn’t the installation and hosting of these apps, but the expertise in using them together. Hosting is sort of a loss leader, or easy entryway. I expect downloaded VMWare & Xen appliances to be a logical progressive step from hosting, or an alternative for more tech-savvy organizations, and on-site full installations to be the greater demand.

In truth I have some trepidation about getting burdened with too much hosting responsibility and being required to spend more time than I’d like administering installations, and less time developing testing solutions.

So is Jumpbox more of a competitor or potential collaborator? Or, does it really just help grow the open source QA tools pie?