Missing the forest for the trees
Understanding customer needs for being able to do good testing sounds like commonplace that doesn’t need a special discussion. And testing projects often omit to discuss them indeed. Testing teams often start their project involvement with reading functional (or non-functional) specifications, skipping the customer view. I’m still mentioning it on the first place in the introduction of approaches to tackle testing complexity.
Imagine that you need to test the database that we discussed in the same article. You got to make choices, which of the possible combinations you test and which of them not, given you cannot realistically cover all of them. Imagine that you know that 98% of your customers are small and mid enterprises with up to 1.000 employees. Naturally, you might conclude that the majority of your customers will prefer:
- small size deployments with 1-2 instances of the database (over large-scale, highly available and distributed database deployments) to save on hardware and software cost
- will prefer to install the software on their own (as opposed to inviting an external vendor to do the deployment) to save on service cost
- will prefer to operate the database deployment in production on their own (as opposed to engaging a support services vendor) to save on service cost
Given these requirements, you might conclude that your testing should focus on 1) quality of small scale deployments, 2) superb installation experience and on 3) ease of serviceability. This is of course an oversimplification of all the factors that one needs to consider in the real world, but I hope it is good enough to make the point.
At the same time, you might not know that the minority of your customers (2% represented by big enterprises) are generating 98% of your revenues. Such a ratio isn’t a rare case in the real world. Now what? This is completely changing the game. High-availability likely becomes a priority. Usability of installation is nice, but perhaps not that important as one would think, because those big customers prefer to employ external vendors, who are experts in doing deployments and who may prefer interfaces that enable automation of large scale deployment over an easy to use graphical, but manual interface. Those customers are also likely to engage an external service vendor (most probably the software vendor itself) to provide support services, so they don’t need to waste time on it. Consequently, testing will focus on completely different targets than you might have thought based on the initial information.
Understanding your customer can make a huge difference, how good or bad you are in testing. You may easily miss the forest for the trees, when jumping right away on analyzing functional specifications or, God forbid, on test automation without understanding your customer. The trick is that it is often difficult to find reliable and usable customer data. Anyway, try to do your best to find them. It’s worth the effort. Try to talk to marketing, support, professional services or any other applicable departments, if data are not readily available. They will not necessarily provide you with hard customer data, they may still be able to give you some soft metrics and a useful perspective on the way your customers use your product.