Test Coverage and Fermentation

Measuring test coverage makes sense - but maybe not in the way you think. Here's why:

Test Automation coverage is like fermentation bubbles

1. Seeing it doesn't mean things are going to be fine
- but not seeing it may indicate that the process itself may not yield good results.

2. could see it and still get an unconsumable product:
Coverage only tells you that code was executed by tests, not whether there are quality risks or issues.

3. When you don't see it on new stuff, this most certainly tells you that you'll have problems:
Your testing process and your development are out of sync, and that's a pretty good indicator that your development process itself is low quality.

It also tells you that the team is definitely not practicing TDD - because just like fermentation bubbles, test coverage is a side product, not a goal, of TDD.

If you ran a pickle factory - what do you think would happen if you set a target for amount of fermentation bubbles?

Focus on the pickles - not the bubbles.
But when there are no bubbles (tests) - you need to investigate.

Why test coverage targets backfire

 Most organizations adopt a "Percent-Test-Coverage" metric as part of their Definition of Done. Very often, it's set by managers, and becomes a target which developers have to meet. This approach is often harmful. And here's why:

The good, bad and ugly of Coverage Targets

Setting coverage targets is often intended as both a means of improving the product, as well as improving the process - and getting developers to think about testing.

The good

It's good that developers begin to pay closer attention to the questions, "How do I test this?" as well as, "Where are tests missing?" Unfortunately, that's about it.

The bad

The first problem is that large legacy code bases starting with zero test coverage will put developers into a pickle: If I work on a component with 0% test coverage, then any amount of tests I write will still keep that number close to Zero. Hence, the necessary compromise becomes not setting a baseline number, just ask for the number to increase. The usually envisioned 80+% targets are visions for a distant future rather than something useful today.

Looking into the practice - as long as the organization is set up to reward minimizing the amount of time invested into unit testing, the outcome will be that developers try to meet the test coverage targets with minimum effort. 

Also, when developers have no experience in writing good tests, their tests may not do what they're supposed to do.

The ugly

There are many ways in which we can meet coverage targets that fulfill Goodhart's Law:

Any metric that becomes a target stops being useful.

Often, developers will feel unhappy producing tests that they create only to meet the target, considering the activity a wasteful addition to the process, which provides no benefit.

At worst, developers will spend a lot of time creating tests that provide a false sense of confidence, which is even worse than knowing that you have no tests to rely on.

But how can this happen? Let's see ...

The first antipattern is, of course, to write tests that check nothing.

Assertion-free testing

Let's take a look at this example:

Production code

int divide(int x, int y) { return x/y; }

Test

print divide(6,2);

When looking at our code coverage metrics, we will see a 100% coverage. But: the test will always pass - even when we break the production code. Only if we were to inspect the actual test output (which is manual effort that we probably won't do) - we will what the test does: There is no automated failure detection, and the tests aren't even written in a way that we would detect a failure: Who would detect the problem if suddenly, we got a "2" instead of a "3" - we don't even know which result would have been correct!

Testing in the wrong place

Look at this example:

Class Number {
int x;
void setX(int val) {x=val;}
void getX() {return x;}
void compute() { x=x?x/x^x:x-x*(x+x); }
}

n = new Number();
n.set(5);
assert (N.x == 5);
assert (N.get() == 5);

In this case, we have a code coverage of 75% - we're testing x, we're testing the setter, and we're testing the getter.

The "only" thing we're not testing is the compute function, which is actually the one place where we would expect problems, where other developers might have questions as to "What does that do, and why are we doing it like that?" - or where various inputs could lead to undesirable outcomes.

Testing wrongly

Take a peek at this simple piece of code:

int inverse (int x) { return 1/x; }
void printInverse(int x) { print "inverse(x);" }

assert (inverse(1) == 1);
assert (inverse(5) == 0);
spyOn(print);
printInverse(5);
assert (print.toHaveBeenCalled());

The issues

There are numerous major problems here, all of which in combination make that test more dangeous than helpful:

Data Quality

We may get a null pointer exception if the method is called without initializing x first, but we neither catch nor test this case.

Unpexpected results

If we feed 0 into the function, we get a Divide by Zero. We're not testing for this, and it will lead to undesired outcomes.

Missing the intent

The inverse function returns 0 for every number other than 1 and -1. It probably doesn't do what it's expected to do. How do we know? Is it poorly named, or poorly implemented?

Testing the wrong things

The print function's output is most likely not what we expect, but our tests still pass.

Conclusion

If we rely on coverage metrics, we might assume that we have 100% test coverage, but in practice, we may have very unreliable software that doesn't even work as intended.

In short: this way of testing tests that the code does what it does, not that it does what it should.

Then what?

The situation can be remedied, but not with numerical quotas. Instead, developers need education on what to test, how to test, and how to test well.

While this is a long topic, this article already shows some extremely common pitfalls that developers can - and need to - steer clear from. Coverage metrics leave management none the wiser: the real issue is hidden behind a smoke screen of existing tests.

Why test coverage tagets backfire

Most organizations adopt a "Percent-Test-Coverage" metric as part of their Definition of Done. Very often, it's set by managers, and becomes a target which developers have to meet. This approach is often harmful. And here's why:

The good, bad and ugly of Coverage Targets

Setting coverage targets is often intended as both a means of improving the product, as well as improving the process - and getting developers to think about testing.

The good

It's good that developers begin to pay closer attention to the questions, "How do I test this?" as well as, "Where are tests missing?" Unfortunately, that's about it.

The bad

The first problem is that large legacy code bases starting with zero test coverage will put developers into a pickle: If I work on a component with 0% test coverage, then any amount of tests I write will still keep that number close to Zero. Hence, the necessary compromise becomes not setting a baseline number, just ask for the number to increase. The usually envisioned 80+% targets are visions for a distant future rather than something useful today.

Looking into the practice - as long as the organization is set up to reward minimizing the amount of time invested into unit testing, the outcome will be that developers try to meet the test coverage targets with minimum effort. 

Also, when developers have no experience in writing good tests, their tests may not do what they're supposed to do.

The ugly

There are many ways in which we can meet coverage targets that fulfill Goodhart's Law:

Any metric that becomes a target stops being useful.

Often, developers will feel unhappy producing tests that they create only to meet the target, considering the activity a wasteful addition to the process, which provides no benefit.

At worst, developers will spend a lot of time creating tests that provide a false sense of confidence, which is even worse than knowing that you have no tests to rely on.

But how can this happen? Let's see ...

The first antipattern is, of course, to write tests that check nothing.

Assertion-free testing

Let's take a look at this example:

Production code

int divide(int x, int y) { return x/y; }

Test

print divide(6,2);

When looking at our code coverage metrics, we will see a 100% coverage. But: the test will always pass - even when we break the production code. Only if we were to inspect the actual test output (which is manual effort that we probably won't do) - we will what the test does: There is no automated failure detection, and the tests aren't even written in a way that we would detect a failure: Who would detect the problem if suddenly, we got a "2" instead of a "3" - we don't even know which result would have been correct!

Testing in the wrong place

Look at this example:

Class Number {
int x;
void setX(int val) {x=val;}
void getX() {return x;}
void compute() { x=x?x/x^x:x-x*(x+x); }
}

n = new Number();
n.set(5);
assert (N.x == 5);
assert (N.get() == 5);

In this case, we have a code coverage of 75% - we're testing x, we're testing the setter, and we're testing the getter.

The "only" thing we're not testing is the compute function, which is actually the one place where we would expect problems, where other developers might have questions as to "What does that do, and why are we doing it like that?" - or where various inputs could lead to undesirable outcomes.

Testing wrongly

Take a peek at this simple piece of code:

int inverse (int x) { return 1/x; }
void printInverse(int x) { print "inverse(x);" }

assert (inverse(1) == 1);
assert (inverse(5) == 0);
spyOn(print);
printInverse(5);
assert (print.toHaveBeenCalled());

The issues

There are numerous major problems here, all of which in combination make that test more dangeous than helpful:

Data Quality

We may get a null pointer exception if the method is called without initializing x first, but we neither catch nor test this case.

Unpexpected results

If we feed 0 into the function, we get a Divide by Zero. We're not testing for this, and it will lead to undesired outcomes.

Missing the intent

The inverse function returns 0 for every number other than 1 and -1. It probably doesn't do what it's expected to do. How do we know? Is it poorly named, or poorly implemented?

Testing the wrong things

The print function's output is most likely not what we expect, but our tests still pass.

Conclusion

If we rely on coverage metrics, we might assume that we have 100% test coverage, but in practice, we may have very unreliable software that doesn't even work as intended.

In short: this way of testing tests that the code does what it does, not that it does what it should.

Then what?

The situation can be remedied, but not with numerical quotas. Instead, developers need education on what to test, how to test, and how to test well.

While this is a long topic, this article already shows some extremely common pitfalls that developers can - and need to - steer clear from. Coverage metrics leave management none the wiser: the real issue is hidden behind a smoke screen of existing tests.

Test Automation Specialists - what a waste!

"Test Automation Engineers require someone who will write the test cases for them, and they can start automating right after the software is deployed onto the test environment."

It's 2022, and I just had this conversation yesterday. Seriously?

Overspecialization leads to Waste

If we work like that, we needlessly create bottlenecks, we test too late, our tests will be of low quality, and we will spend a lot of time on identifying, tracking and reworking defective deliveries. We produce a lot of work, but not quality.

In a well-functioning software organization, there can be no such role as a "test automation engineer" who is fed work by "test designers." 

Although it sounds "resource efficient," that approach generates massive amounts of waste in all categories of Lean Muda: 

Defects

When automated tests aren't written by developers, and only available once a software package is released, we end up with three kinds of defects:

1. Product defects, i.e. things the product should (or shouldn't) do, but that weren't correctly implemented, which require another software update.
2. Automation defects, i.e. things the test automation should (or shouldn't) do, but that weren't correctly implemented. Until clarified, these lead to significant communication overhead.
3. False negatives, i.e. inadequate automation not catching existing defects. This is significantly more likely when automation isn't written close to the code, and might omit dangerous cliffs in the product implementation.

Handover

Whenever actors in a process change, there's a handover. At the point of handover, we induce delay, lose information and risk the introduction of defects. This makes our process slow and error-prone. When test itself becomes a source of error, we should stop and think.

Motion

Handovers also mean that we're moving artifacts: test descriptions, build versions and defects - without need. We also need to coordinate this motion, which means we induce additional meetings into our process, which also slows us down and adds non-value added overhead. Motion also leads to inventory waste.

Inventory

As previously mentioned, un-automated tests will pile up as inventory, as will un-tested deliveries. When tests fail, the defect notifications are made available at a point in time when developers are working on other things. Hence, we are adding three queues to our process. Each queue contains work waiting to be done, and the combined length of all these queues is value delay for our customers.

Waiting

As mentioned in my article on CI/CD industry benchmarks, the time required between when a developer makes a change, and when a developer receives feedback on whether everything is correct shouldn't exceed a few minutes, preferably just a few seconds. If automation begins upon delivery, this period of time is impossible to warrant. The delivered product has to wait a significant amount of time until it can produce customer value, and developers will turn their attention elsewhere.

Overprocessing

Having different people automate tests and write productive code leads to "overprocessing," often called "gold-plating:" As mentioned in my article about the test pyramid, we should try to automate as close to the code as possible. When we're doing high-level tests which could instead be low-level tests, we make our test suite more complex, slower and less reliable than possible.Unit tests are extremely quick both to write and execute. Test automation engineers who automate black box/gray box tests will spend a significantly higher amount of work to write tests with poorer outcomes.

Overproduction

Since the amount of labor to create a product increment in server code and test automation are extremely hard to balance, the strategy of automating after the fact usually leads to developed product features piling up, waiting to be automated. As a consequence, the decision is often to release these without proper test automation coverage to flush the waiting queue. This builds up "technical debt" which deteriorates product sustainability.

Human potential

When developing software with multiple people, we act as a team. We can learn from each other, and we can align: Someone who can automate tests, can also write features. Someone who can write features, can also automate. As a team, we can align during planning "who does what," and fill our information gaps so that those doing the work have full information and don't need anyone else to fill in for them. Techniques like pair programming or code reviews ensure that even if one person makes a mistake, someone else will catch it before a defect even enters the system. It's a limiting belief to think that somehow people can't learn - we all can, if the conditions are favorable. Maybe not everyone will be the greatest developer, but we don't need to: we work within our ability, and grow our capacity a little every day.

To confine a test automation engineer to automating defined test specifications, and to confine coders to write code without understanding how to do that without defects - is an insult to our human intellect, and a very expensive one, at that:  All the above seven wastes cost a lot of money that we could save if we would just improve our collaboration, communication and skillset.

Lean Software Test Automation

Please, let me clarify. I don't believe that test automation expertise is a waste - the waste is actually hidden in the process, and how we use our talent.

A much leaner approach which avoids all of the above eight wastes is to educate and empower - use QA to help developers learn how to define the right tests, how to apply processes like Test-Driven Development and Behaviour-Driven Development, and guide them on their journey to deliver Zero Defect Quality. And in reverse, having developers upskill our test automation experts to help them deliver integrated pieces of value including working software.

The purpose of test automation in such an organization isn't to find defects - it's to maintain the high frequency of delivery of high quality products, and to serve as a living documentation for all behaviours of the product which is maintained as part of the development process without overhead.

Tests without code knowledge are waste

I start with an outrageous claim: "Software tests made without knowledge of the code are waste."

Now, let me back that claim up with an example - a calculator applet which you can actually use:

Calculator

0

Have fun, play around with this little applet ... it works if your JS is enabled!

Now let us create a test strategy for this applet:

Black Box test strategy

Let's discuss how you would test this, by going into the list of features this applet offers:

• Basic maths: Addition, Subtraction, Multiplication, Division
• Parentheses and nested parentheses
• Comparisons: Less, Equal, Greater ...
• Advanced maths: Exponents, Square Root, Logarithm, Maximum, Minimum, Rounding (prefixed by "Math.")
• Trigonometry: Sinus, Cosinus, Tangens, Arc-Functions (also prefixed)
• Variables: defining, referencing, modifying
• Nested functions: combining any set of aforementioned functionality
• And a few more ...

Wow - I'm sure you can already see where our test case catalogue is going with minimal coverage, and we haven't even considered edge cases or negative tests yet!

How many things could go wrong in this thing? Twenty, thirty? Fifty? A thousand?

How many tests would provide an appropriate test coverage? Five, ten, a hundred?

How much effort is required for all this testing? 

Would it shock you if I said ...

All those tests are a waste of time!

Frankly speaking, I would test this entire thing with a single test case, because everything beyond that is just waste. 

I am confident that I can do that because I know the code:

  <div class="row">
	<div class="card"
             style="margin-top:1rem;
                        margin-left:2rem;
                        margin-right:2rem; width: 18rem;
                        box-shadow: 4px 4px 4px 1px rgba(0, 0, 0, 0.2);">
	  <b class="card-header text-center bg-primary text-light">
            Calculator</b>
	  <div class="card-body">
	  	<textarea class="form-control"
                          id="calc_input"
                          rows="1" cols="20"
                          style="text-align: right;"
		          oninput="output.innerHTML = eval(this.value)"
                          ></textarea>
	  </div>
	  <div class="card-footer text-right">
		  <b id="output">0</b>
	  </div>
	</div>
  </div>
  

Yes, you see this right. The entire code is just look-and-feel. There is only a single executable statement: "eval(this.value)", so we don't have a truckload of branch, path, line, statement coverage and whatnotever that we need to cover:
All relevant failure opportunities are already covered by JavaScript's own tests for its own eval() function, so why would we want to test it again? 

The actual failure scenarios

Seriously, this code has only the following opportunities for failure:

• Javascript not working (in this case, no test case will run anyways)
• Accidentally renaming the "output" field (in this case, all tests will fail anyways)
• User Input error (not processed)
• Users not knowing how to use certain functions
  (which is a UX issue ... but how relevant?)

Without knowing the code, I need to test an entire catalogue of test cases.

Knowing and understanding the code, I would  reduce the entire test to a single Gherkin spec:

The only relevant test

Given: I see the calculator

When: I type "1+0".

Then: I see "1" as computation result.

So why wouldn't we want to test anything else?
Because: the way the code is written, if this test passes, then all other tests we could think of would also pass.

But why not write further tests?

A classical BDD/TDD approach would mandate us to incrementally define all application behaviours, create a failing test for each, then add passing code, then refactor the application. 

If we would do this poorly, we would really end up creating hundreds of tests and writing explicit code to pass each of them - and that's actually a problem with incremental design unaware of the big picture!

The point is that we wrote the minimum required code to meet the specified functionality right from the start: code that has only a single failure opportunity (not being executed) - and after having this code in place, there's no way we can write another failing test that meets the feature specifications!

And that's why a close discussion between testers and developers is essential in figuring out which tests are worthwhile and which aren't.

The importance of testablility

In the article on Black Box Testing, we took a look at the testing nightmare caused by a product that was not designed for testing. This led to a followup point: "Testability", which is also a quality attribute of the ISO:25010. Let us examine a little closer what the value of testability for product development actually is.

What is even a "test"?

In science, every hypothesis needs to be falsifiable, i.e. it must be logically and practically possible to find counter-examples. 

Why are counter-examples so important? Let's use a real-world scenario, and start with a joke.

A sociologist, a physisicst and a mathematician ride on a train across Europe. As they cross the border into Switzerland, they see a black sheep. The sociologist exclaims, "Interesting. Swiss sheep are black!". The physicist corrects, "Almost. In Switzerland, there are black sheep." The mathematician shakes her head, "We only know that in Switzerland, there exists one sheep that is black on at least one side."

• The first statement is very easy to disprove: they just need to encounter a sheep that isn't black.
• The second statement is very hard to disprove: because even if the mathematician were right and another angle would reveal the sheep to not be fully black, the statement weren't automatically untrue, because there could be other sheep somewhere in Switzerland that are black.
• Finally, the third statement holds true, because the reverse claim ("There is no sheep that is black on one side in Switzerland") has already been disproven by evidence.

This leads to a few follow up questions we need to ask about test design.

Test Precision

Imagine that our test setup to verify the above statements looked like this:

1. Go to the travel agency.
2. Book a flight to Zurich.
3. Fly to Zurich.
4. Take a taxi to the countryside.
5. Get out at a meadow.
6. Walk to the nearest sheep.
7. Inspect the sheep's fur.
8. If the sheep's fur color is black, then Swiss sheep are black.

Aside from the fact that after running this test, you might be stranded in the Winter Alps at midnight wearing nothing but your pyjamas and you're a couple hundred Euros poorer, this test could go wrong in so many ways.

For example, your travel agency might be closed. Or, you could have insufficient funds to book a flight. You could have forgotten your passport and aren't allowed to exit the airport, you could go to a meadow that has cows and not sheep, and the sheep's fur inspection might yield fleas. Which of these has anything to do with whether Swiss sheep are black?

We see this very often in "black box tests" as well:

• it's very unclear what we're actually trying to test
• just because the test failed, that doesn't mean that the thing we wanted to know is untrue
• there's a huge overhead cost associated with validating our hypothesis
• we don't return to a "clean slate" after the test
• Success doesn't provide sufficient evidence to verify the test hypothesis.

Occam's Razor

In the 13th century, a monk by the name of Occam came up with what's known today as Occam's Razor, i.e., "entities should not be multiplied without necessity". Taking a look at our above test, that would mean that every test step that has nothing to do with sheep or fur color should be removed from the design.

The probability to run this test successfully increases by isolating the test object (sheep's fur) as far as possible, and eliminating all variability from the test that isn't directly associated to the hypothesis itself.

Verifyability

We verify a hypothesis by finding a counter-example to what's called the "alternate hypothesis" and assuming that if this one is untrue, then its logical opposite, called the "null hypothesis" is true. Unfortunately, this means we have to think in reverse.

In our example: To prove that all sheep are black, we have to sample all sheep. That's difficult. It's much easier to sample one non-black sheep, and thereby falsify that all sheep are black. If we fail to produce even one, then all sheep must be black.

Repeatability and Reproducibility

A proper test is a systematic, preferrably repeatable and reproducible, way of verifying our hypothesis. That means, it should be highly predictable in its outcome, and we should be able to test as often as we want.

Again, going back to our example, if we design our test like this:

1. Take a look at a Swiss sheep.
2. If it is white, then sheep are not black.
3. If sheep are not - not black, then sheep are black.

This is terrible test design, because of some obvious flaws in each step: 

1. The setup is an uncontrolled random sample. Since sheep are white or black, running this test on an unknown setup, means we haven't ruled out anything if we picked a black sheep.
2. The alternate hypothesis is incomplete: Should the sheep be brown, then it is also not white.
3. Assuming that step 2 didn't trigger, we would conclude that brown = black.

Since the "take a look at a Swiss sheep" is already part of the test design, each time we repeat this test, we get a different outcome, and we can't reproduce anything either, because if I run this test, my outcome will be different from yours.

Reproducibility

A repeatability problem occurs when the same setup can generate different results. In our example, "take a look at" could be fixed, assuming we take the mathematician's advice, by re-phrasing step 1 to: "Look at a Swiss sheep from all angles." This would lead everyone to reach the same conclusion.

We might also have to define what we call "white" and "black", or whether we would classify "brown" as a kind of white.

We increase reproducibility by being precise on how we examine the object under test, and which statements we want to make about our object under test.

Repeatability

Depending on what the purpose of our test is, we are doing well by removing the variation in our object under test. So, if our test objective is to prove or falsify that all sheep are black, we can set up a highly repeatable, highly reproducible test like this:

1. Get a white Swiss sheep.
2. Identify the color of the sheep.
3. If it is not black, then the statement that Swiss sheep are black is false.

This experiment setup is going to produce the same outcome for everyone conducting the test anywhere across the globe, at any point in time.

While there is a risk that we fail in step 1 (if we can't get hold of a Swiss sheep), we could substitute the test object with a picture of a Swiss sheep without affecting the validity of the test itself.

What is testability?

A good test setup has:

• a verifiable test hypothesis
• a well-defined test object
• a precise set of test instructions
• absolutely minimized test complexity
• high repeatability and reproducibility

When all these are given, we can say that we have good testability. The more compromises we need to make in any direction, the worse our testability gets. 

A product that has high testability allows us to formulate and verify any relevant test hypothesis with minimal effort.

A product with poor testability has a high difficulty associated with formulating or verifying a test hypothesis. This difficulty might translate into an increase of any or all of the following:

• complexity
• effort
• cost
• duration
• validity
• uncertainty

In conclusion

The more often you want to test a hypothesis, the more valuable high testability becomes.

With increasing change frequency, the need to re-verify a formerly true hypothesis also increases. 

Design your product from day 1 to be highly testable.

By the time you discover that a product's testability is unsustainably low, it's often extremely expensive to notch it up to the level where you need it.

The problem of Black Box Tests

One of the most fundamental enablers of agile ways of working is the ability to swiftly and reliably detect problems in your product - that is, to test it efficiently and effectively. Unfortunately, the "traditional" approach of black-box testing a running application is hardly useful for this purpose. 

I have created an executable use case to illustrate the problem. 

Let's take a look at this little service, and assume that it would be your responsibility to test it in a way that you can reliably tell whether it works correctly, or which problems it has:

You can call this service in the mini applet included on this page if you have JavaScript enabled..

Alea iacta est

You need Javascript to run this demo.

Yes, just try it out - roll the dice!
That's about as simple as an application can get.
Can you imagine how much effort it takes to properly test this simple application?

It's not enough to roll the dice once and get a number between 1 and 6 - how do you know that there isn't a possibility that the application might generate results outside that range?

And how would you know that you have fair dice? Call the service a thousand times and assume that you would get an approximately even distribution of values? What would be your thresholds for assuming that the dice are "fair"? What if 5% or fewer, or 25% or more results go to one number, which is statistically still possible with a decent probability?

You see the difficulty already.

But let's make this more difficult:

Hello

You need Javascript to run this demo.

What if I told you that this is a call to the same service?
Yes, exactly - you didn't know everything the service does when you created your test concept before.
There's a different feature hidden in the service: if you pass a user name to the request, it will greet you!

This adds a whole new dimension to the test complexity: you have to test with - and without - a user name. And would you want to try different user names?

  But that's not everything:

You lose!

You need Javascript to run this demo.

Did you even catch that this one behaves different?
What if I told you that this is another call to the same service?
Yes, exactly - you still didn't know everything the service does when you created your test concept.
There's another different feature hidden in the service: you can load the dice and cheat!

If you tell the service to cheat you, you will get unfair dice.

So, now you need to run your entire test set from above twice again - with and without cheating.

And we haven't even looked into whether there are multiple ways of cheating, or whether the cheating function always triggers correctly when the variable is set (hint: it doesn't). Good luck without knowing the application where the malfunction is.

But we're not done yet:

I win  

You need Javascript to run this demo.

Did you catch the difference here?

What if I told you that this is yet another call to yet again same service?

There's yet another different feature hidden in the service: if I use my name to cheat, I will get loaded dice in my advantage!

By now, you're probably doubting whether you understood the application at all when you started testing it.

The code

Now - let me blow your mind and tell you how little source code was required to totally blow your test complexity and effort out of proportion:

That's it. This little snippet of code is entirely sufficient to keep a Black Box tester busy for hours, potentially days, and still remain unable to make a reliable statement on whether they missed anything, and which problems the product may or may not have.

Depending on how your application is designed, a few minutes of development effort can generate a humongous mountain of effort in testing.

 

And that's why you can't possibly hope to ever achieve a decent test coverage on an application without knowing the code.

Testability

There's another problem: this code wasn't written with testing in mind (or, much rather: purposely written with poor testability in mind -- hee hee) so you have no way of ever coming up with an easier way to test this service, until it's rewritten.

And that's why you can't maintain sustainable high quality unless developers and testers actively collaborate to build highly testable software that is easy to work with, both for software changes and testing. 

Think of minimizing sustainable lead time - consider the total effort from request to release, and consider it both for initial creation and future modification. There's no point in optimizing for development speed if you slow down testing more than that, and likewise, there's no point in delivering minimal code if the consequence is totally bloated test scope.

Otherwise, you'll not be very agile.

Test Cocooning

 "How do you deal with Legacy code that lacks test coverage?" - even miniscule small changes are hazardous, and often, a necessary rewrite is postponed forever because it's such a nightmare to work with. Even if you have invested time into your test coverage after taking over the system, chances are there are still parts of the system you need to deal with that aren't covered at all. So this is what I propose in this situation:

Test Cocooning is a reversed TDD cycle, and it should be common sense.

The Cocooning process

Test Cocooning is a pretty straightforward exercise: 

1. Based on what you think the code does, you create a cocooning test.
1. If the test fails, you didn't understand the code correctly and you have to improve your test.
2. If the test passes, you have covered a section of the code with a test that ensures you don't accidentally break the tested aspect of the code.
2. Based on what you think the code does, you make a breaking change.
1. If the test fails in the way you thought it would, you have a correct understanding of that piece of code.
2. If the test passes, you didn't understand the code correctly and you have to improve your test (back to step 1)
3. Intermediate activity: Of course, you revert the change to restore the behaviour that you have covered with test.
3. Within the scope of your passing test, you begin to improve:
1. Create lower-levelled tests that deal with more specifics of the tested code (e.g. unit tests.)
2. Refactor based on the continuous and recurrent execution of all the relevant tests.
3. Refactor your tests as well.
4. Re-run the original cocooning test to ensure you didn't mess up anywhere!

Once a cocooning cycle is completed, you should have reworked a small section of your Legacy code to be Clean(er) Code that is more workable for change.

Iterating

You may need to complete multiple cocooning cycles until you have a sufficient amount of certainty that you can work on the code reliably.

Backtracking

The important secret of successful Test Cocooning is that you need to backtrack both on the code and your tests - after completing all relevant cocooning cycles, you'll need to re-run:

• your cocooning tests against the original legacy code. 
• your unrefactored original cocooning tests against the new code.
• your unrefactored original cocooning tests against the original legacy code.

Yes, that's painful and a lot of overhead, but it's your best bet in the face of dangerous, unworkable code, and believe me - it's a lot less painful than what you'll experience when some nasty bugs slip through because you skipped any of these.

Working Cocooned code

Once you have your test cocoon, you can work the cocooned code - only within the scope of your cocoon - to fix bugs and to build new features.

Bugfixes

Fixing bugs relies on making a controlled breach to your cocoon.

Metaphorically speaking, you need to be like a spider that caught a bug and sucks it dry before discarding the woven husk.

1. Create a test cocoon for the current behaviour which passes under the current faulty(!) conditions of the code segment that exactly reproduces the bug as though it were desired behaviour.
2. Create a test which fails due to the bug, i.e. add a second test that exactly reverses the cocooned behaviour.
3. Write the code that meets the requirement of the failing test.
1. As a consequence, the cocooned passing test for the bug should now fail.
2. Ensure that no other tests have failed.
3. If another test has failed, ensure that this is intentional.
4. Eliminate the broken cocoon test that reproduces the bug's behaviour.
1. If there were other tests that failed, now is the time to modify these tests one by one.
5. Backtrack like described above to ensure that nothing slipped. 

Modifying features

Modifying existing behaviour should be treated exactly like a bugfix.

New functionality

If you plan to add new functionality to Legacy Code, your best bet is to develop this code in isolation  from the cocooned legacy and only communicate via interfaces, ensuring that the cocoon doesn't break. 

When you really need to invoke new code from the Legacy, treat the modification like a bugfix.

Rewrite

A rewrite should keep the cocoon intact. Don't cling to any of the Legacy code and consider your cocooning efforts "sunken cost" - otherwise, you risk reproducing the same mess with new statements. 

Closing remarks

1. I believe that test cocooning requires both strong test and development expertise, so if you have different specialists on your team, I would highly recommend to build the cocoon in pairing.
2. Cocoon tests are often inefficient and have poor performance. You do not need to add these tests to your CI/CD pipeline. What you must add to your pipeline is the lower-level tests that replicate the unit behaviour of the cocoon. It's totally sufficient to rerun the cocoon tests when you work on the cocooned Legacy segment.
3. Cocooning is a workaround for low-quality code. When time permits, rewrite it with Clean Code and you can discard the cocoon along with the deleted code.
4. Do not work on Legacy Code without a solid Cocoon. The risk outweighs the effort.

Test Coverage Matrix

Whether you're transitioning towards agile ways of working on a Legacy platform or intend to step up your testing game for a larger system developed in an agile fashion, at some point, it pays to set up a Coverage Matrix to see where it pays to invest effort - and where it doesn't.

Before we start

First things first: the purpose of an agile Coverage Matrix isn't the same as a traditional project-style coverage matrix that's mostly concerned with getting the next release shipped. I don't intend to introduce a mechanism that adds significant overhead with little value, but to give you a means of starting the right discussions at the right time and to help you think in a specific direction. Caveat emptor: It's up to you to figure out how far you want to go down each of the rabbit holes. "Start really simple and incrementally improve" is good advice here!

What I'm proposing in this article will sound familiar to the apt Six Sigma practitioner as a simplified modification of the method, "Quality Function Deployment." And that's no coincidence.

Coverage Characteristics

Based on the ISO/IEC 9126, quality characteristics can be grouped into Functionality, Reliability, Usability, Efficiency, Maintainability and Portability. These are definitely good guidance. 

To simplify matters, I like to start the initial discussion by labelling the columns of the matrix:

• Functionality ("Happy Cases")
• Reliability ("Unhappy Cases")
• Integration
• Performance
• Compliance
• UX

Of course, we can clarify a lot more on what each of these areas means, but let's provide some leeway for the first round of discussion here. The most important thing is that everyone in the room has an aligned understanding on what these are supposed mean. 

If you are in the mood for some over-engineering, add subcategories for each coverage characteristic, such as splitting Performance into "efficiency", "speed", "scalability", "stress resilience" etc. That will bloat up the matrix and may make it more appropriate to flip rows and columns on the matrix.

Test Areas

Defining test areas falls into multiple categories, which correlate to the "Automation Test Pyramid". 

• User journeys
• Data flow
• Architectural structure
• Code behaviour

There are other kinds of test areas, such as validation of learning hypotheses around value and human behaviour, but let's ignore these here. Let's make a strong assumption that we know what "the right thing" is, and we just want to test that "we have things right." Otherwise, we'd open a can of worms here. You're free to also cover these, adding the respective complexity.

Functional areas

In each test area, you will find different functional areas, which strongly depend on what your product looks like.

User journeys

There are different user journeys with different touchpoints how your user interacts with your product. 

For example, a simple video player app might have one user flow for free-to-play, another for registration, another for premium top-up, and another for GDPR compliant deregistration as well as various flows such as "continue to watch my last video" or "download for offline viewing". These flows don't care what's happen technically.

Data flow

Take a look at how the data flows through your system as certain processes get executed. Every technical flow should be consistent end-to-end.

For example, when you buy a product online, the user just presses "Purchase", and a few milliseconds later, they get a message like "Thank you for your order." The magic that happens inbetween is make or break for your product, but entirely irrelevant for the user. In our example, that might mean that the system needs to make a purchase reservation, validate the user's identity and their payment information, must conduct a payment transaction, turn the reservation into an order, ensure the order gets fulfilled etc. If a single step in this flow breaks, the outcome could be an economic disaster. Such tests can become a nightmare in microservice environments where they were never mapped out.

Architectural structure

Similar to technical flow, there are multiple ways in which a transaction can occur: it can happen inside one component (e.g. frontend rendering), it can span a group of components (e.g. frontend / backend / database) or even a cluster (e.g. billing service, payment service, fulfilment service) and in the worst case, multiple ecosystems consisting of multiple services spanning multiple enterprises (e.g. Google Account, Amazon Fulfilment, Salesforce CRM, Tableau Analytics).

In architectural flow, you could list the components and their key partner interfaces. For example:

• User Management
• CRM API
• DWH API
• Payment 
• Order API
• Billing API

Architectural flow is important in the sense that you need to ensure that all relevant product components and their interactions are covered.

You can simplify this by first listing the relevant architectural components, and only drilling down further if you have identified a relevant hotspot.

Code behaviour

At the lowest level is always the unit test, and different components tend to have different levels of coverage - are you testing class coverage, line coverage, statement coverage, branch coverage - and what else? Clean Code? Suit yourself.

Since you can't list every single behaviour of the code that you'd want to test for without turning a Coverage Matrix into a copy of your source code, you'll want to focus on stuff that really matters: do we think there's a need to do something?

Bringing the areas together

There are dependencies between the areas - you can't have a user flow without technical flow, you won't have technical flow without architectural flow, and you won't have architectural flow without code behaviour. Preferably, you don't need to test for certain user flows at all, because the technical and architectural flows already cover everything. 

If you can relate the different areas with each other, you may learn that you're duplicating or missing on key factors.

Section Weight

For each row, for each column, assign a value on how important this topic is. 

For example, you have the user journey "Register new account." How important do you think it's to have the happy path automated? Do you think the negative case is also important? Does this have impact on other components, i.e. would the user get SSO capability across multiple products? Can you deal with 1000 simultaneous registrations? Is the process secure and GDPR compliant? Are users happy with their experience?

You will quickly discover that certain rows and columns are "mission critical", so mark them in red. Others will turn out to be "entirely out-of-scope", such as testing UX on a backend service, so mark them gray. Others will be "basically relevant" (green) or "Important" (yellow).

As a result, you end up with a color-coded matrix.

The key discussion that should happen here is whether the colors are appropriate. An entirely red matrix is as unfeasible as an entirely gray matrix.

A sample row: Mission critical, important, relevant and irrelevant

Reliability Level

As the fourth activity, focus on the red and yellow cells and take a look at a sliding scale on how well  you're doing in each area and assign a number from 0 to 10 with this rough guidance:

• 0 - We're doing nothing, but know we must.
• 3 - We know that we should to more here.
• 5 - We've got this covered, but with gaps.
• 7 - We're doing okay here.
• 10 - We're using an optimized, aligned, standardized, sustainable approach here.

As a consequence, the red and yellow cells should look like this:

A sample matrix with four weighted fields.

As you would probably guess by now, the next step for discussion would be to look at the big picture and ask, "What do we do with that now?"

The Matrix

Row Aggregate

For each row, figure out what the majority of colors in that row is, and use that as the color of the row. Next, add up all the numbers. This will give you a total number for the row. 

This will give you an indicator which row is most important to address - the ones in red, with the lowest number.

The Row Aggregate

Column Aggregate

You can use the same approach for the columns, and you will discover which test type is covered best.  I would be amazed if Unhappy Path or Compliance turn out to have poor coverage when you first do this exercise, but the real question is again: Which of the red columns has the lowest number?

The Column aggregate

After conducting all the above activities, you should end up with a matrix that looks similar to this one:

A coverage matrix

Working with the Matrix

There is no "The right approach" to whether to work on improving coverage for test objects or test types - the intent is to start a discussion about "the next sensible thing to do," which totally depends on your specific context.  

As per our example, the question of "Should we discuss the badly covered topic Performance which isn't the most important thing, or should we cover the topic of architectural flow?" has no correct answer - you could end up with different groups of people working hand in hand to improve both of these, or you could focus on either one.

How-To Use

You can facilitate discussions with this matrix by inviting different groups of interest - business people, product people, architects, developers, testers - and start a discussion on "Are we testing the right things right, and where or how could we improve most effectively?"

Modifications 

You can modify this matrix in whatever way you think: Different categories for rows or columns, drill-in, drill-across - all are valid.

For example, you could have a look at only functional tests on user journeys and start listing the different journeys, or you could explicitly look at different types of approaching happy path tests (e.g., focusing on covering various suppliers, inputs, processing, outputs or consumers)

KISS 

This method looks super complicated if you list out all potential scenarios and all potential test types - you'd take months to set up the board, without even having a coversation. Don't. First, identify the 3-4 most critical rows and columns, and take the conversation from there. Drill in only when necessary and only where it makes sense.

Why there's no traditional test strategy in agile organizations

The test strategy is a fundamental pillar of quality assurance. It would seem plausible that quality is independent of development approach, hence the same approach can be used irrespective of whether  we are Agile or not.

Nothing could be further from the truth. Indeed, a traditional approach to quality assurance is entirely incompatible with an agile approach - to the point where it becomes a problem in terms of performance and quality! Hence, if you have a traditional test management background and enter an agile environment, there are a few things you need to understand, lest your best intentions be met with massive resistance and pushback.

The goal of testing

Why do we test? This fundamentally changes between a traditional and an agile setting:

From accepting the past to sustaining the future

Traditional, stage-gated testing is centered around the idea that at some point, the work previously completed by developers enters into a QA process, during which the conformity of a product towards specified requirements is assured.  This is past-oriented. It assumes that the product / release / version is "finished".

An agile tester works with a team delivering in small increments - in the case of Continuous Deployment, that could be hundreds of increments per day. The product is never "finished". There will always be more work in the future. The agile team is supposed to work in a way that whatever was the last delivery, we can call it a day, and tomorrow, we won't find a shambles caused by whatever we messed up.

The testing mission

Let's start with the shift in testing objective, because different goals verify a different approach:

From "finding defects" to "preventing defects"

Traditional testing assumes that software has defects, and that developers make mistakes. In the worst circumstances, it assumes that unless checked upon, developers will not do what they are told.
Traditional QA serves a threefold purpose:

• Verify that developers have done what they claim to have done
• Catch the mistakes in the work of developers
• Discover the defects in the software

In an agile setting, this is reframed, to a much more positive outlook on both the people and their work. Our agile testing serves three purposes:

• Consistently deliver high, preferrably zero-defect, quality
• Provide evidence that the product does the right thing
• Prevent defects as early as possible

 As a consequence, some core concepts of how we go about "Agile Testing" change:

Test activity

The major contribution of testing changes:

From mitigating project risk to enabling teams

Whereas a traditional Test Strategy contributes to the project objectives, mitigating the risk of project failure through poor quality, agile testers enable their teams to continuously and consistently work in an environment where quality related risks are a non-issue.
You have to think of it like this: A nuclear power plant has to invest effort into preventing nuclear fallout. A watermill doesn't do that, because it wouldn't even make sense.

Likewise, an agile test strategy won't concern itself with defects. Hence, a lot of things you may have learned that are "must-have" as a test manager are just plain irrelevant:

Test Preparation

We move on to the first major area of a test strategy: preparation. Traditionally, we create a test case catalog by writing test cases to cover the content of the Specification documentation. Then, during the test phase, execute the test cases to verify conformity to requirements. If a test case finds no defect, we label it as passed. Once a certain threshold of test cases were passed, we can give a "Go" from testing.

There's one fundamental problem here when working with agile teams: there is no specification document! Then what? To make a long story short, we still have a specification, and we still have test cases: The tests are the specification! 

A few things don't exist in an agile organization, though:

Test Case Catalog

The Test Case Catalog is built on the idea that there is something like a major delivery that needs to be tested successfully  to meet the project objectives. That idea is incompatible with agile ways of working.
On a high level, we discriminate two types of tests: those that ensure quality, and those that help us understand what quality is.
All tests of the first category become part of our test suite - they are run on every build, they get run as soon as they get created, and they get created as soon as the feature starts being developed.

There is no test case catalog that has been created "upfront, for later execution".

Risk-Based Testing

Typically, a test case catalog contains a myriad of test cases that the team will not have time to conduct. Hence, Risk-Based testing helps to match capacity with effort, while reducing the overall quality risk. In an agile organization, things look different.
We don't develop things that don't need to pass tests. And we don't create tests that don't need to pass. Testing is part of development, passing tests is part of the process, and the tests are as much part of the product as the productive code itself.

Test Data Creation

Most traditional testers have at some point encountered the difficults of acquiring the necessary data to conduct certain test cases - sometimes, it's not entirely clear what data is needed, how it should look like and (in case of mass data) where to obtain it. When techniques like BDD with Specification by Example are in use, we have our test data as part of the product design.

Test scenario setup

In traditional software testing, it would often take hours, sometimes days, to set up an intricate test scenario required to check only one thing in the software. And then pray that we didn't make a mistake - or all that effort was lost! If that's the case, then our architecture has some issue: Tests in our pipeline should bring everything necessary to run as quickly as possible - in seconds rather than hours! And if a scenario takes days to prepare, it'll be a maintenance nightmare, so we'd rather not have any of these to begin with.

Test scenarios move from a per-release basis to a per-code-change basis, which means that it doesn't even make sense to plan scenario setup strategically: it moves entirely to the work execution level.

Defect Management

Traditional test managers feel appalled when an agile team tells them that they neither have, nor want, defect management. Given the typical defect rates observed in traditional Waterfall organizations, it's unthinkable to not systematize and institutionalize defect management.

Let me return to the nuclear plant example. Of course, it needs to have both processes and means to deal with toxic waste: You'd get second thoughts if there was no hazardous waste management. But what if there were barrels labelled as "Nuclear waste" in your local sushi diner? You'd bolt for the door - because such a thing simply doesn't belong there!
It's the same for defects. They don't belong in an agile organization. That's why we don't need defect management.

And with defect management, we lose the need for many other things that would be part of a good traditional test strategy:

Defect management process

In an agile team, dealing with non-conformance is easy: When a test turns red, the developer stops what they're doing, fixes the problem, and continues.
Under ideal circumstances, this takes seconds - if it takes minutes, it may already be an issue where they involve other people on the team. That's it.

Defect prioritization

Don't we all have fun with the arguments that ensue around whether a defect is Priority 1,2 or 3? We don't need any meetings to align and agree on a priority model are pointless if there's a "stop the line" process where any defect would immediately interrupt all other work until resolved. 

Defect status model

Given that a known issue is either someone's top priority being worked on, or it's already fixed, we don't need much of a status model. That reduces organizational complexity by a massive amount.

Defect Tracking

There is nothing to track, by default. If there are defects to track, we have problems we shouldn't be having.

Defect management tool

The agile organization would prefer to resolve the root cause that would mandate the need for such a tool. We should not institiute a tool based on the idea that quality problems are inevitable.

Defect status meetings

No defects, no defect meeting.

Defect reports

What would you expect from a report with no data?

Defect KPIs

Who hasn't seen the ping pong that ensues when a defect was shoved between developer and tester a dozen times, with the tester claiming "it's a defect", and the developer arguing it isn't? When you measure testers against rejected defects while measuring developers against the amount of defects, you generate this confliect. Without defect-related KPIs, there's no such conflict.

Test Management

It's an unfair assertion to say that there's no test management, because agile tests are well-managed.

Test Plans

What we don't want is assigning and scheduling test cases or types to individual testers, irrespective of whether a feature will actually be delivered. Instead, every backlog item has all necessary tests related to it, It's clearly defined who runs them (the CI/CD pipeline), where (on the CI/CD stage) and when (on every build). Part of the refactoring process would be to move the test plan away from a backlog item into the test suite - a default element of the test plan becomes the "full regression" of everything that was formerly built. Hence, a traditional test plan becomes redundant.

Test Tracking

Once you've got your test case catalog, you need to track your test cases. Not so in an agile setting, where the CI/CD pipeline runs and monitors every single test case. "If the test exists and is part of the suite, it will be run every time we change the code, and the developer will receive a notification if anything is wrong." - what would you want to track?

Test Documentation

This isn't fair, because a test documentation exists: in the log files of the CI/CD pipeline, for every single change to the codebase. It's is just that we don't give a hoot about the documentation of test individual cases, because the entire document would read "Step - executed, passed. Test - executed, passed", since wherever that's not true, we get information on what wasn't okay, when and where.

Test Reporting

We don't do stuff like reporting the percentage of test cases passed, failed and "not-run". There are only two valid conditions for our entire software: "all tests passed", or "tests not passed".  And there's not really a need to report testing at all, because if a single test hasn't passed, we can't deploy. So, we really only need to report development progress.

Test Status Meetings

In a waterfall organization, we need to track test status, typically by having routine meetings where testers report their progress Is vs Should, the amount of defects they found, and how many of them were already closed, plus an estimate how likely they consider completion of their work by the end of the test period.
This meeting wouldn't make any sense in an agile organization, because there would be nothing to talk about.

Test Management Suite

Agile organizations rely heavily on automation. There's probably a tool for everything that is relevant and can be automated. Still, you're not going to find a Test Management or Application Lifecycle Management Suite - because it has nothing to do.
If your test cases written in the central repository and managed by the pipeline, your test protocols are managed by your artifactory, and you don't have any defects to track ... what exactly would you expect such a tool to do?

Roles and Responsibility

We need to agree on which role has which responsibility in the testing process - who writes test cases, who reviews and approves them, who runs them, who communicates defects, who tracks them, and so on. None of this would be required in an agile setting: The team writes, reviews and runs test cases, and deals with any problems encountered. The role is called, "Agile team member", and the responsibility is "contribute to the team's success." What that means can be more or less flexible, and just like the different members in a family have different strengths and weaknesses, we don't want the game of "Not my responsibility" or "Why didn't you..." - because none of these discussions help us reach our team goals. The only discussion we are looking for is "How can I contribute to ..." - and that may change upon need. We wouldn't want a static document to contradict what people feel they can achieve.

Test Levels

We have a Test Pyramid, and technically, that doesn't change in an agile environment. But it means a different thing than in a traditional organization. 

In a traditional organization, we would decide up front on certain test levels, which tests to run on which level, and when to do these test levels.

In agile development, the test levels are fluid. We decide on a test, and we execute it. We then refactor it, to conduct it on the most effective level, and that should - first and foremost, be the unit level. Pulling every test to the lowest possible level is essential to retaining a sustainable test suite, and that means there can be no hard cut of what to do where.

Test types

We have the "Test Quadrants" which give a simple and clear overview of what should be tested, and whether it's automatable. Unlike a Test Strategy document, which would define once for all which of these test types we use, what we do to cover them and where to run them. In an agile setting, these quadrants are more of a constant discussion trigger - "do more of this, maybe some less of that is enough, how can we do better here ..."

Test Automation

Automation is often an afterthought in classical test strategies - we identify a number of critical test cases that need to become part of the Regression suite, and as far as effort permits, we automate them. This wouldn't work in an agile setting. With the "automate everything" DevOps mentality, we wouldn't define what to automate - we'd automate everything that needs to be done frequently, and that would include pretty much all functional tests. It would also include configuration, data generation and scraping data from the system. We wouldn't include it in a test strategy, though, because how helpful is a statement of "we do what makes sense" - as if anything to the contrary would make sense.

Release Management

Ideally, we would be on a Continuous Deployment principle - and where that's not feasible, it should be Continuous Delivery. We also want to have a "Release on Demand" principle, that is: when something is developed, it should be possible to deploy this content without delay. Whether it would be released to users immediately or after a period should be a business, and not a technical, question. In most settings, the content would already be live and using "Dark Release" mechanisms, become available without changes to the code base.

Test Phases

A major concern in traditional testing is the coordination of the different test phases required to meet the launch date. Some activities, like test preparation, need to be completed before the delivery of the software package, and all activities need to be completed a few days before launch to meet the project schedule.
When the team is delivering software many times an hour, and wants to deploy to Production at least once a day, you're going to be short on luck creating a phase gated schedule - acceptance, integration and system tests happen in parallel, continuously. They don't block each others, and they take minutes rather than weeks.

Scheduling non-functional tests

Yes, there are some tests, like Pen-Tests or load tests, that we wouldn't run on every build.
Whereas a traditional test strategy would put these on a calendar with clear test begin/end periods, we'd schedule intervals or triggers for these types of test - for example, "nightly, weekly, quarterly" or "upon every major build or change to the environment".

Installation schedule

A major issue in traditional testing is the schedule for installation - what will be installed, when, and by whom. We'd prefer to "push" builds through the automated pipeline, and expect every member of the team to "pull" any version of the software onto an environment of their choice at any time.
If you think this results in chaos, try reframing this into, "How would the software and the work need to look like for this to not result in chaos?" - it's possible!

Go/No-Go Decisions

In an agile setting, you're not working to achieve a "Go" approval from management by providing evidence that quality is sufficient. If there are any quality issues, we would not continue anyways.

Test Coverage

A key metric in traditional testing is test coverage - both the percentage of requirements covered with tests, and the percentage of tests successfully completed. Neither of these statements makes sense in a setting where a requirement is defined by tests, and where the work isn't "done" until the test is successfully completed: test coverage, from the perspective of a traditional definition, must always be 100% by default. Why then measure it?

Restricted approval

Whereas the traditional tester is usually tasked with doing whatever is required to ensure that a new software release can get the "Go" approval, the approval is usually made "with restrictions". That means: "we know the delivery isn't really up to the mark, but we have a deadline, and can't afford to postpone it." Everyone knows the quality isn't there, it's just a question of how many corners we can cut, and by how much.  Agile testers have a different goal: Understanding that corners we did cut today will need to be smoothed out in the future - requires us to ensure that there are no cut corners!

Approval without restrictions

When there are no cut corners, there is no "restricted Go", and when all approval decisions are always an unconditional, and unanimously "Go" - there is no need for Go/No-Go decisions.

Test Environments

Probably one of the hardest battles fought when transitioning from a traditional testing approach to an agile testing approach is the topic of environments: more environments add complexity, constraints on environments reduce agility. The fewer environments we have, the better for us.

Environment configuration

If we move towards a DevOps approach, we should also have "infrastructure as code". Whereas a traditional test team would usually have one specialist to take care of environment configuration, we'd expect the config to be equal, or at least, equivalent, to the Production environment - with no manual configuration activity. Our test strategy should be able to rely on our CI/CD pipeline to be able to bootstrap a test environment in minutes.

Installation schedule

A major issue in traditional testing is the schedule for installation - what will be installed, when, and by whom. We'd prefer to "push" builds through the automated pipeline, and expect every member of the team to "pull" any version of the software onto an environment of their choice at any time.
If you think this results in chaos, try reframing this into, "How would the software and the work need to look like for this to not result in chaos?" - it's possible!


After all these, I hear your cry, ... but ...

How about Regulatory Compliance?

"We need evidence for, e.g. SOX Compliance, to warrant that the software meets specification, that all tests were executed, by whom, when, and with which outcome." Yes. That could be. And that's no conflict to agile testing.
I would even claim that agile testing is a hundred times better suited to meet regulatory requirements than traditional testing. And here's why:

1. The exact statements which were used to do a test are code. 
2. That means, they have no space for interpretation, and can be repeated and reproduced infinitely.
3. It also means they are subject to version control. We can guarantee that the test result is exactly correlated to the test definition at that timestamp. Our test cases are tamper-proof.
4. There is no room for human error in execution or journaling. The results are what they are and mean what they mean. 
5. All test runs are protocolled exactly as defined. There is no way for any evidence to be missing. By storing test results in an artifactory, we have timestamped, tamper-proof evidence. And tons of it.

Proper agile testing as nothing to be afraid of when facing an audit. It's better prepared than traditional test management could ever be, and under most circumstances, an auditor would require much less evidence to ascertain compliance than an agile team would require just to do their job.


All that said, this begs the question ...

Do we need an Agile Test Strategy?

If you've been paying close attention to all I've written above, you may wonder if there's a need for a test strategy in an agile environment at all.
After all of the above points, the answer will surprise you: Yes, indeed we need an agile test strategy.
We will explore this test strategy in more detail at another time. At this time, let me just reduce this to headlines of what it means and will cover:

The Agile Test Strategy ...

• belongs to the teams developing software.
• is a living documentation that explains what is currently happening.
• focuses on increasing customer satisfaction and product value.
• is closely related to the Definition of Done, the objective standard of when a team has completed their work on any given item.
• contains the team's quality-related Working Agreements that provide a explanation of how the team is collaborating to meet quality objectives.
• addresses organizational and technological measures required to attain and sustain high quality.
• minimizes both current and future total effort required to deliver sustainable high quality.
• optimizes the Test Pyramid
• utilizes the Test Quadrants
• leverages manual testing to optimize learning

The key question an agile test strategy would always revolve around is, "What are we doing to meet our commitment to quality with even less effort?"

Conclusion

With this definitely long article, I hope that I could shed some light into the differences between agile and traditional testing strategy. You can add massive value to an agile organization if they don't have a test strategy yet, and you can always find something within an agile test strategy that can be optimized. What you should not do, however, is to consider the absence of a traditional test strategy as a "flaw" in an agile organization.

You need to be familiar with the differences in the two approaches, so that you can avoid making unhelpful suggestions and focus your efforts on the things which move the teams forward.


The major assumption of this article is that an agile organization is actually committed to and using the engineering practices required to attain continuous high quality. Where this isn't true, the Agile Test Strategy would include the way to achieve this condition - it wouldn't focus on instituting practices or mechanisms contrary to this goal.