Tests without code knowledge are waste

I start with an outrageous claim: "Software tests made without knowledge of the code are waste."

Now, let me back that claim up with an example - a calculator applet which you can actually use:

Calculator

0

Have fun, play around with this little applet ... it works if your JS is enabled!

Now let us create a test strategy for this applet:

Black Box test strategy

Let's discuss how you would test this, by going into the list of features this applet offers:

• Basic maths: Addition, Subtraction, Multiplication, Division
• Parentheses and nested parentheses
• Comparisons: Less, Equal, Greater ...
• Advanced maths: Exponents, Square Root, Logarithm, Maximum, Minimum, Rounding (prefixed by "Math.")
• Trigonometry: Sinus, Cosinus, Tangens, Arc-Functions (also prefixed)
• Variables: defining, referencing, modifying
• Nested functions: combining any set of aforementioned functionality
• And a few more ...

Wow - I'm sure you can already see where our test case catalogue is going with minimal coverage, and we haven't even considered edge cases or negative tests yet!

How many things could go wrong in this thing? Twenty, thirty? Fifty? A thousand?

How many tests would provide an appropriate test coverage? Five, ten, a hundred?

How much effort is required for all this testing? 

Would it shock you if I said ...

All those tests are a waste of time!

Frankly speaking, I would test this entire thing with a single test case, because everything beyond that is just waste. 

I am confident that I can do that because I know the code:

  <div class="row">
	<div class="card"
             style="margin-top:1rem;
                        margin-left:2rem;
                        margin-right:2rem; width: 18rem;
                        box-shadow: 4px 4px 4px 1px rgba(0, 0, 0, 0.2);">
	  <b class="card-header text-center bg-primary text-light">
            Calculator</b>
	  <div class="card-body">
	  	<textarea class="form-control"
                          id="calc_input"
                          rows="1" cols="20"
                          style="text-align: right;"
		          oninput="output.innerHTML = eval(this.value)"
                          ></textarea>
	  </div>
	  <div class="card-footer text-right">
		  <b id="output">0</b>
	  </div>
	</div>
  </div>
  

Yes, you see this right. The entire code is just look-and-feel. There is only a single executable statement: "eval(this.value)", so we don't have a truckload of branch, path, line, statement coverage and whatnotever that we need to cover:
All relevant failure opportunities are already covered by JavaScript's own tests for its own eval() function, so why would we want to test it again? 

The actual failure scenarios

Seriously, this code has only the following opportunities for failure:

• Javascript not working (in this case, no test case will run anyways)
• Accidentally renaming the "output" field (in this case, all tests will fail anyways)
• User Input error (not processed)
• Users not knowing how to use certain functions
  (which is a UX issue ... but how relevant?)

Without knowing the code, I need to test an entire catalogue of test cases.

Knowing and understanding the code, I would  reduce the entire test to a single Gherkin spec:

The only relevant test

Given: I see the calculator

When: I type "1+0".

Then: I see "1" as computation result.

So why wouldn't we want to test anything else?
Because: the way the code is written, if this test passes, then all other tests we could think of would also pass.

But why not write further tests?

A classical BDD/TDD approach would mandate us to incrementally define all application behaviours, create a failing test for each, then add passing code, then refactor the application. 

If we would do this poorly, we would really end up creating hundreds of tests and writing explicit code to pass each of them - and that's actually a problem with incremental design unaware of the big picture!

The point is that we wrote the minimum required code to meet the specified functionality right from the start: code that has only a single failure opportunity (not being executed) - and after having this code in place, there's no way we can write another failing test that meets the feature specifications!

And that's why a close discussion between testers and developers is essential in figuring out which tests are worthwhile and which aren't.