The problem of Black Box Tests

One of the most fundamental enablers of agile ways of working is the ability to swiftly and reliably detect problems in your product - that is, to test it efficiently and effectively. Unfortunately, the "traditional" approach of black-box testing a running application is hardly useful for this purpose. 

I have created an executable use case to illustrate the problem. 

Let's take a look at this little service, and assume that it would be your responsibility to test it in a way that you can reliably tell whether it works correctly, or which problems it has:

You can call this service in the mini applet included on this page if you have JavaScript enabled..

Alea iacta est

You need Javascript to run this demo.

Yes, just try it out - roll the dice!
That's about as simple as an application can get.
Can you imagine how much effort it takes to properly test this simple application?

It's not enough to roll the dice once and get a number between 1 and 6 - how do you know that there isn't a possibility that the application might generate results outside that range?

And how would you know that you have fair dice? Call the service a thousand times and assume that you would get an approximately even distribution of values? What would be your thresholds for assuming that the dice are "fair"? What if 5% or fewer, or 25% or more results go to one number, which is statistically still possible with a decent probability?

You see the difficulty already.

But let's make this more difficult:

Hello

You need Javascript to run this demo.

What if I told you that this is a call to the same service?
Yes, exactly - you didn't know everything the service does when you created your test concept before.
There's a different feature hidden in the service: if you pass a user name to the request, it will greet you!

This adds a whole new dimension to the test complexity: you have to test with - and without - a user name. And would you want to try different user names?

  But that's not everything:

You lose!

You need Javascript to run this demo.

Did you even catch that this one behaves different?
What if I told you that this is another call to the same service?
Yes, exactly - you still didn't know everything the service does when you created your test concept.
There's another different feature hidden in the service: you can load the dice and cheat!

If you tell the service to cheat you, you will get unfair dice.

So, now you need to run your entire test set from above twice again - with and without cheating.

And we haven't even looked into whether there are multiple ways of cheating, or whether the cheating function always triggers correctly when the variable is set (hint: it doesn't). Good luck without knowing the application where the malfunction is.

But we're not done yet:

I win  

You need Javascript to run this demo.

Did you catch the difference here?

What if I told you that this is yet another call to yet again same service?

There's yet another different feature hidden in the service: if I use my name to cheat, I will get loaded dice in my advantage!

By now, you're probably doubting whether you understood the application at all when you started testing it.

The code

Now - let me blow your mind and tell you how little source code was required to totally blow your test complexity and effort out of proportion:

That's it. This little snippet of code is entirely sufficient to keep a Black Box tester busy for hours, potentially days, and still remain unable to make a reliable statement on whether they missed anything, and which problems the product may or may not have.

Depending on how your application is designed, a few minutes of development effort can generate a humongous mountain of effort in testing.

 

And that's why you can't possibly hope to ever achieve a decent test coverage on an application without knowing the code.

Testability

There's another problem: this code wasn't written with testing in mind (or, much rather: purposely written with poor testability in mind -- hee hee) so you have no way of ever coming up with an easier way to test this service, until it's rewritten.

And that's why you can't maintain sustainable high quality unless developers and testers actively collaborate to build highly testable software that is easy to work with, both for software changes and testing. 

Think of minimizing sustainable lead time - consider the total effort from request to release, and consider it both for initial creation and future modification. There's no point in optimizing for development speed if you slow down testing more than that, and likewise, there's no point in delivering minimal code if the consequence is totally bloated test scope.

Otherwise, you'll not be very agile.