Why there's no traditional test strategy in agile organizations

The test strategy is a fundamental pillar of quality assurance. It would seem plausible that quality is independent of development approach, hence the same approach can be used irrespective of whether  we are Agile or not.

Nothing could be further from the truth. Indeed, a traditional approach to quality assurance is entirely incompatible with an agile approach - to the point where it becomes a problem in terms of performance and quality! Hence, if you have a traditional test management background and enter an agile environment, there are a few things you need to understand, lest your best intentions be met with massive resistance and pushback.

The goal of testing

Why do we test? This fundamentally changes between a traditional and an agile setting:

From accepting the past to sustaining the future

Traditional, stage-gated testing is centered around the idea that at some point, the work previously completed by developers enters into a QA process, during which the conformity of a product towards specified requirements is assured.  This is past-oriented. It assumes that the product / release / version is "finished".

An agile tester works with a team delivering in small increments - in the case of Continuous Deployment, that could be hundreds of increments per day. The product is never "finished". There will always be more work in the future. The agile team is supposed to work in a way that whatever was the last delivery, we can call it a day, and tomorrow, we won't find a shambles caused by whatever we messed up.

The testing mission

Let's start with the shift in testing objective, because different goals verify a different approach:

From "finding defects" to "preventing defects"

Traditional testing assumes that software has defects, and that developers make mistakes. In the worst circumstances, it assumes that unless checked upon, developers will not do what they are told.
Traditional QA serves a threefold purpose:

• Verify that developers have done what they claim to have done
• Catch the mistakes in the work of developers
• Discover the defects in the software

In an agile setting, this is reframed, to a much more positive outlook on both the people and their work. Our agile testing serves three purposes:

• Consistently deliver high, preferrably zero-defect, quality
• Provide evidence that the product does the right thing
• Prevent defects as early as possible

 As a consequence, some core concepts of how we go about "Agile Testing" change:

Test activity

The major contribution of testing changes:

From mitigating project risk to enabling teams

Whereas a traditional Test Strategy contributes to the project objectives, mitigating the risk of project failure through poor quality, agile testers enable their teams to continuously and consistently work in an environment where quality related risks are a non-issue.
You have to think of it like this: A nuclear power plant has to invest effort into preventing nuclear fallout. A watermill doesn't do that, because it wouldn't even make sense.

Likewise, an agile test strategy won't concern itself with defects. Hence, a lot of things you may have learned that are "must-have" as a test manager are just plain irrelevant:

Test Preparation

We move on to the first major area of a test strategy: preparation. Traditionally, we create a test case catalog by writing test cases to cover the content of the Specification documentation. Then, during the test phase, execute the test cases to verify conformity to requirements. If a test case finds no defect, we label it as passed. Once a certain threshold of test cases were passed, we can give a "Go" from testing.

There's one fundamental problem here when working with agile teams: there is no specification document! Then what? To make a long story short, we still have a specification, and we still have test cases: The tests are the specification! 

A few things don't exist in an agile organization, though:

Test Case Catalog

The Test Case Catalog is built on the idea that there is something like a major delivery that needs to be tested successfully  to meet the project objectives. That idea is incompatible with agile ways of working.
On a high level, we discriminate two types of tests: those that ensure quality, and those that help us understand what quality is.
All tests of the first category become part of our test suite - they are run on every build, they get run as soon as they get created, and they get created as soon as the feature starts being developed.

There is no test case catalog that has been created "upfront, for later execution".

Risk-Based Testing

Typically, a test case catalog contains a myriad of test cases that the team will not have time to conduct. Hence, Risk-Based testing helps to match capacity with effort, while reducing the overall quality risk. In an agile organization, things look different.
We don't develop things that don't need to pass tests. And we don't create tests that don't need to pass. Testing is part of development, passing tests is part of the process, and the tests are as much part of the product as the productive code itself.

Test Data Creation

Most traditional testers have at some point encountered the difficults of acquiring the necessary data to conduct certain test cases - sometimes, it's not entirely clear what data is needed, how it should look like and (in case of mass data) where to obtain it. When techniques like BDD with Specification by Example are in use, we have our test data as part of the product design.

Test scenario setup

In traditional software testing, it would often take hours, sometimes days, to set up an intricate test scenario required to check only one thing in the software. And then pray that we didn't make a mistake - or all that effort was lost! If that's the case, then our architecture has some issue: Tests in our pipeline should bring everything necessary to run as quickly as possible - in seconds rather than hours! And if a scenario takes days to prepare, it'll be a maintenance nightmare, so we'd rather not have any of these to begin with.

Test scenarios move from a per-release basis to a per-code-change basis, which means that it doesn't even make sense to plan scenario setup strategically: it moves entirely to the work execution level.

Defect Management

Traditional test managers feel appalled when an agile team tells them that they neither have, nor want, defect management. Given the typical defect rates observed in traditional Waterfall organizations, it's unthinkable to not systematize and institutionalize defect management.

Let me return to the nuclear plant example. Of course, it needs to have both processes and means to deal with toxic waste: You'd get second thoughts if there was no hazardous waste management. But what if there were barrels labelled as "Nuclear waste" in your local sushi diner? You'd bolt for the door - because such a thing simply doesn't belong there!
It's the same for defects. They don't belong in an agile organization. That's why we don't need defect management.

And with defect management, we lose the need for many other things that would be part of a good traditional test strategy:

Defect management process

In an agile team, dealing with non-conformance is easy: When a test turns red, the developer stops what they're doing, fixes the problem, and continues.
Under ideal circumstances, this takes seconds - if it takes minutes, it may already be an issue where they involve other people on the team. That's it.

Defect prioritization

Don't we all have fun with the arguments that ensue around whether a defect is Priority 1,2 or 3? We don't need any meetings to align and agree on a priority model are pointless if there's a "stop the line" process where any defect would immediately interrupt all other work until resolved. 

Defect status model

Given that a known issue is either someone's top priority being worked on, or it's already fixed, we don't need much of a status model. That reduces organizational complexity by a massive amount.

Defect Tracking

There is nothing to track, by default. If there are defects to track, we have problems we shouldn't be having.

Defect management tool

The agile organization would prefer to resolve the root cause that would mandate the need for such a tool. We should not institiute a tool based on the idea that quality problems are inevitable.

Defect status meetings

No defects, no defect meeting.

Defect reports

What would you expect from a report with no data?

Defect KPIs

Who hasn't seen the ping pong that ensues when a defect was shoved between developer and tester a dozen times, with the tester claiming "it's a defect", and the developer arguing it isn't? When you measure testers against rejected defects while measuring developers against the amount of defects, you generate this confliect. Without defect-related KPIs, there's no such conflict.

Test Management

It's an unfair assertion to say that there's no test management, because agile tests are well-managed.

Test Plans

What we don't want is assigning and scheduling test cases or types to individual testers, irrespective of whether a feature will actually be delivered. Instead, every backlog item has all necessary tests related to it, It's clearly defined who runs them (the CI/CD pipeline), where (on the CI/CD stage) and when (on every build). Part of the refactoring process would be to move the test plan away from a backlog item into the test suite - a default element of the test plan becomes the "full regression" of everything that was formerly built. Hence, a traditional test plan becomes redundant.

Test Tracking

Once you've got your test case catalog, you need to track your test cases. Not so in an agile setting, where the CI/CD pipeline runs and monitors every single test case. "If the test exists and is part of the suite, it will be run every time we change the code, and the developer will receive a notification if anything is wrong." - what would you want to track?

Test Documentation

This isn't fair, because a test documentation exists: in the log files of the CI/CD pipeline, for every single change to the codebase. It's is just that we don't give a hoot about the documentation of test individual cases, because the entire document would read "Step - executed, passed. Test - executed, passed", since wherever that's not true, we get information on what wasn't okay, when and where.

Test Reporting

We don't do stuff like reporting the percentage of test cases passed, failed and "not-run". There are only two valid conditions for our entire software: "all tests passed", or "tests not passed".  And there's not really a need to report testing at all, because if a single test hasn't passed, we can't deploy. So, we really only need to report development progress.

Test Status Meetings

In a waterfall organization, we need to track test status, typically by having routine meetings where testers report their progress Is vs Should, the amount of defects they found, and how many of them were already closed, plus an estimate how likely they consider completion of their work by the end of the test period.
This meeting wouldn't make any sense in an agile organization, because there would be nothing to talk about.

Test Management Suite

Agile organizations rely heavily on automation. There's probably a tool for everything that is relevant and can be automated. Still, you're not going to find a Test Management or Application Lifecycle Management Suite - because it has nothing to do.
If your test cases written in the central repository and managed by the pipeline, your test protocols are managed by your artifactory, and you don't have any defects to track ... what exactly would you expect such a tool to do?

Roles and Responsibility

We need to agree on which role has which responsibility in the testing process - who writes test cases, who reviews and approves them, who runs them, who communicates defects, who tracks them, and so on. None of this would be required in an agile setting: The team writes, reviews and runs test cases, and deals with any problems encountered. The role is called, "Agile team member", and the responsibility is "contribute to the team's success." What that means can be more or less flexible, and just like the different members in a family have different strengths and weaknesses, we don't want the game of "Not my responsibility" or "Why didn't you..." - because none of these discussions help us reach our team goals. The only discussion we are looking for is "How can I contribute to ..." - and that may change upon need. We wouldn't want a static document to contradict what people feel they can achieve.

Test Levels

We have a Test Pyramid, and technically, that doesn't change in an agile environment. But it means a different thing than in a traditional organization. 

In a traditional organization, we would decide up front on certain test levels, which tests to run on which level, and when to do these test levels.

In agile development, the test levels are fluid. We decide on a test, and we execute it. We then refactor it, to conduct it on the most effective level, and that should - first and foremost, be the unit level. Pulling every test to the lowest possible level is essential to retaining a sustainable test suite, and that means there can be no hard cut of what to do where.

Test types

We have the "Test Quadrants" which give a simple and clear overview of what should be tested, and whether it's automatable. Unlike a Test Strategy document, which would define once for all which of these test types we use, what we do to cover them and where to run them. In an agile setting, these quadrants are more of a constant discussion trigger - "do more of this, maybe some less of that is enough, how can we do better here ..."

Test Automation

Automation is often an afterthought in classical test strategies - we identify a number of critical test cases that need to become part of the Regression suite, and as far as effort permits, we automate them. This wouldn't work in an agile setting. With the "automate everything" DevOps mentality, we wouldn't define what to automate - we'd automate everything that needs to be done frequently, and that would include pretty much all functional tests. It would also include configuration, data generation and scraping data from the system. We wouldn't include it in a test strategy, though, because how helpful is a statement of "we do what makes sense" - as if anything to the contrary would make sense.

Release Management

Ideally, we would be on a Continuous Deployment principle - and where that's not feasible, it should be Continuous Delivery. We also want to have a "Release on Demand" principle, that is: when something is developed, it should be possible to deploy this content without delay. Whether it would be released to users immediately or after a period should be a business, and not a technical, question. In most settings, the content would already be live and using "Dark Release" mechanisms, become available without changes to the code base.

Test Phases

A major concern in traditional testing is the coordination of the different test phases required to meet the launch date. Some activities, like test preparation, need to be completed before the delivery of the software package, and all activities need to be completed a few days before launch to meet the project schedule.
When the team is delivering software many times an hour, and wants to deploy to Production at least once a day, you're going to be short on luck creating a phase gated schedule - acceptance, integration and system tests happen in parallel, continuously. They don't block each others, and they take minutes rather than weeks.

Scheduling non-functional tests

Yes, there are some tests, like Pen-Tests or load tests, that we wouldn't run on every build.
Whereas a traditional test strategy would put these on a calendar with clear test begin/end periods, we'd schedule intervals or triggers for these types of test - for example, "nightly, weekly, quarterly" or "upon every major build or change to the environment".

Installation schedule

A major issue in traditional testing is the schedule for installation - what will be installed, when, and by whom. We'd prefer to "push" builds through the automated pipeline, and expect every member of the team to "pull" any version of the software onto an environment of their choice at any time.
If you think this results in chaos, try reframing this into, "How would the software and the work need to look like for this to not result in chaos?" - it's possible!

Go/No-Go Decisions

In an agile setting, you're not working to achieve a "Go" approval from management by providing evidence that quality is sufficient. If there are any quality issues, we would not continue anyways.

Test Coverage

A key metric in traditional testing is test coverage - both the percentage of requirements covered with tests, and the percentage of tests successfully completed. Neither of these statements makes sense in a setting where a requirement is defined by tests, and where the work isn't "done" until the test is successfully completed: test coverage, from the perspective of a traditional definition, must always be 100% by default. Why then measure it?

Restricted approval

Whereas the traditional tester is usually tasked with doing whatever is required to ensure that a new software release can get the "Go" approval, the approval is usually made "with restrictions". That means: "we know the delivery isn't really up to the mark, but we have a deadline, and can't afford to postpone it." Everyone knows the quality isn't there, it's just a question of how many corners we can cut, and by how much.  Agile testers have a different goal: Understanding that corners we did cut today will need to be smoothed out in the future - requires us to ensure that there are no cut corners!

Approval without restrictions

When there are no cut corners, there is no "restricted Go", and when all approval decisions are always an unconditional, and unanimously "Go" - there is no need for Go/No-Go decisions.

Test Environments

Probably one of the hardest battles fought when transitioning from a traditional testing approach to an agile testing approach is the topic of environments: more environments add complexity, constraints on environments reduce agility. The fewer environments we have, the better for us.

Environment configuration

If we move towards a DevOps approach, we should also have "infrastructure as code". Whereas a traditional test team would usually have one specialist to take care of environment configuration, we'd expect the config to be equal, or at least, equivalent, to the Production environment - with no manual configuration activity. Our test strategy should be able to rely on our CI/CD pipeline to be able to bootstrap a test environment in minutes.

Installation schedule

A major issue in traditional testing is the schedule for installation - what will be installed, when, and by whom. We'd prefer to "push" builds through the automated pipeline, and expect every member of the team to "pull" any version of the software onto an environment of their choice at any time.
If you think this results in chaos, try reframing this into, "How would the software and the work need to look like for this to not result in chaos?" - it's possible!


After all these, I hear your cry, ... but ...

How about Regulatory Compliance?

"We need evidence for, e.g. SOX Compliance, to warrant that the software meets specification, that all tests were executed, by whom, when, and with which outcome." Yes. That could be. And that's no conflict to agile testing.
I would even claim that agile testing is a hundred times better suited to meet regulatory requirements than traditional testing. And here's why:

1. The exact statements which were used to do a test are code. 
2. That means, they have no space for interpretation, and can be repeated and reproduced infinitely.
3. It also means they are subject to version control. We can guarantee that the test result is exactly correlated to the test definition at that timestamp. Our test cases are tamper-proof.
4. There is no room for human error in execution or journaling. The results are what they are and mean what they mean. 
5. All test runs are protocolled exactly as defined. There is no way for any evidence to be missing. By storing test results in an artifactory, we have timestamped, tamper-proof evidence. And tons of it.

Proper agile testing as nothing to be afraid of when facing an audit. It's better prepared than traditional test management could ever be, and under most circumstances, an auditor would require much less evidence to ascertain compliance than an agile team would require just to do their job.


All that said, this begs the question ...

Do we need an Agile Test Strategy?

If you've been paying close attention to all I've written above, you may wonder if there's a need for a test strategy in an agile environment at all.
After all of the above points, the answer will surprise you: Yes, indeed we need an agile test strategy.
We will explore this test strategy in more detail at another time. At this time, let me just reduce this to headlines of what it means and will cover:

The Agile Test Strategy ...

• belongs to the teams developing software.
• is a living documentation that explains what is currently happening.
• focuses on increasing customer satisfaction and product value.
• is closely related to the Definition of Done, the objective standard of when a team has completed their work on any given item.
• contains the team's quality-related Working Agreements that provide a explanation of how the team is collaborating to meet quality objectives.
• addresses organizational and technological measures required to attain and sustain high quality.
• minimizes both current and future total effort required to deliver sustainable high quality.
• optimizes the Test Pyramid
• utilizes the Test Quadrants
• leverages manual testing to optimize learning

The key question an agile test strategy would always revolve around is, "What are we doing to meet our commitment to quality with even less effort?"

Conclusion

With this definitely long article, I hope that I could shed some light into the differences between agile and traditional testing strategy. You can add massive value to an agile organization if they don't have a test strategy yet, and you can always find something within an agile test strategy that can be optimized. What you should not do, however, is to consider the absence of a traditional test strategy as a "flaw" in an agile organization.

You need to be familiar with the differences in the two approaches, so that you can avoid making unhelpful suggestions and focus your efforts on the things which move the teams forward.


The major assumption of this article is that an agile organization is actually committed to and using the engineering practices required to attain continuous high quality. Where this isn't true, the Agile Test Strategy would include the way to achieve this condition - it wouldn't focus on instituting practices or mechanisms contrary to this goal.

Entrepreneurial Value for in-house development

"We can't measure the monetary value of a Feature". A common complaint, and oftentimes mere ignorance. It's economically disastrous for an organization that spends money on software development!

Here are a few simple, effective ways of making the value of development transparent even when developing complex in-house systems:

Core goals

Before we start, we need to understand that there are fundamentally two different reasons for developing software. While occasionally, these goals can be combined, they are usually distinct, and one of them takes priority.

Revenue increase

Some components allow the company to generate more revenue - acquire new customers, sell additional goods, etc. For those components, measuring feature value is very straightforward: The value is the difference in revenue that can be attributed to the implementation of a feature.

In these cases, the value of a feature can be found on the "Assets" section of the balance sheet - the difference between old and new assets.

Expense reduction

For established processes, most of the work is aimed at increasing efficiency through performance optimization - saving time or resources (e,g., packaging / fuel etc.) to reduce expenses.

In these cases, the value of a feature can be found on the "Liabilities" section of the balance sheet - the difference between old and new liability.

Equity generation?

Argubably, there may be features that turn out to neither generate revenue nor cut down on expenses.

Many people argue that "you got what you got", and proceed to treat such features as assets - claiming that they are becoming shareholder equity that doesn't show up as cash flow.
I, myself, would argue that this is not the case.

From a technical perspective, such features need to be candidates for removal - in software, every feature has code, and every line of code increases the product's complexity, and complexity correlates to the amount of work required to deliver future value -- hence, such features, though seemingly innocent, are technical liabilities!

Determine feature value

After the above is established, we will not discriminate anymore whether a feature's value is determined by an increase in revenue or reduced expenses. Instead of providing specific techniques or methods, this section focuses on thinking patterns that help in determining the value of feature.
These patterns can be selected or combined based on circumstance.

Value Stream thinking

Understand the operational value stream you are contributing to, and the place you hold in that value stream. Your contribution to that value stream is the leverage you have for development.
For example, when your value stream produces a revenue of $10000 a day, and after the implementation of the feature, it's still $10000 a day ... how much did the feature add? Maybe it reduced operating expenses. If, however, they are still the same, the feature did zilch.

Theory of Constraints is a great method of figuring out whether it's even theoretically possible for your feature to add value: if you're not working on the value stream's current or next constraint, chances are slim that you will add value to the organization!

Make of Buy thinking

There's an economics proverb, "If it's known, and you can earn money with it, someone is already doing it." In many cases, there's already a vendor solution that does the same thing you would be developing, and that vendor has a price tag attached to their solution.

While you still can't know if that's going to be the actual value of your feature, the NPV is capped at whatever this vendor is asking. So, for example, if you could buy a cloud solution that solves your problem at $199 a month, we ignore discount rates and cash flow, calculating NPV for a 5-year period, we'd end up with the feature being worth no more than $12k. So unless you can deliver it cheaper, you may not want to build it.

Service thinking

If you remember the beginning of the Internet age, AOL was a service born when companies realized that they had capacities available that others would pay for. Internet giants like Google and Amazon have since successfully replicated the model of selling a solution to their own problem to others as well. You already paid for it - so every cent you make with a sale is net profit! What is stopping you from capitalizing on the idea? If you're doing it right, there's even a chance you can make more money off selling the feature to others than the value it has within your own organization! More than one software company was borne out of scratching one's own itch.

Even if you're producing something "in-house", always ask the question, "Who else has this problem, and how much are they willing to pay to get it solved?" - if the answer boils down to, "Nobody" or "Nothing", then chances are that you're solving the wrong problem.

Key concepts every Product Owner must understand

Let's take a look at some key concepts every Product Owner should understand:

Manage options

As Product Owner, you receive ideas for things that your product could do. Each of them is an option of something that could be done.

There's usually a flood of ideas, some are better, others worse. You need to sort them through, typically by ordering them in a backlog.

You choose the ideas: Good ideas, bad ideas - big ideas, small ideas. Your call.
As far as possible, avoid promises to anyone - they reduce your freedom of choice.

Each of those ideas requires you to make a certain investment in order to turn it into a Product.
That investment is, first and foremost, money. You need to secure this investment, lest your idea dies before it is realized. That's funding.

Spend to Gain

As Product Owner, you have a development team doing the work of making ideas happen. It's not important for you how they do this. That's what they are experts for. What matters for you: what they work on, and in which order.

Your team has a finite capacity - how much work they can do per time. If you feed more ideas to the team than their capacity, they won't get all of them done. So choose wisely how you want to use this capacity.

Development work is fuelled by money - salary, infrastructure, you name it. The rate at which your team spends money is your burn rate. There's a one-way correlation between burn rate and money - a slight reduction in burn rate often significantly reduces capacity, while an increase in burn rate often has no predictable influence on capacity.

With a stable team, your burn rate is constant. Once you know your capacity and burn rate, you can figure out an approximate investment required for an idea. Your estimates will usually be wrong, and  a better crystal ball will only reduce your capacity.

Outcome Focus

The Return on Invest is how much money you made off realizing the idea. Oftentimes, it's elusive (you can't really know at all), and usually it's entire guesswork before you're done. So keep the investment low until you actually see this Return on Invest and turned an assumption into hard cash.

Look for small product increments that allow you to generate a constant flow of value.
Don't get blinded by big numbers: It's better to get $100 every month, starting tomorrow - than to get $5000 once, two years from now. There are many reasons for that - the biggest one being that you can't know if you'll get that money until you did.

Entrepreneurship

You must compare your Return on Invest (ROI) with your Investment. Let's assume it's your own business. You'd understand immediately that if the ROI is lower than your investment, and if your cash flow is worse than your burn rate - it's just a matter of time until you have to foreclose.

And that's exactly what a Product Owner needs to do:

1. to ensure that the product generates a positive Return on Invest, by eliminating bad ideas and prioritizing good ones.
2. to ensure that there's a positive cash flow by feeding a constant stream of ideas that can be realized with a controlled investment before making a "pivot or persevere" (build something else or more of the same) decision 
3. to prioritize ideas against one another, by figuring out which ones promise the best Return on Invest

Your job

First and foremost, you have to understand money to be a Product Owner.

The development work is the experts' job. Even on prioritizing and formulating ideas, they could be the experts. 

As long as the ideas the development team realizes result in an overall positive cash flow, the team can continue and grow - the business is healthy. And that's your job. Everything else, you can delegate.

A fictional scenario

To employed Product Owners, I like to give this thought experiment:

Imagine that you were on a bazaar.

1. You can buy the product for the investment required.
2. You get the product delivered after the development time has elapsed.
3. You can sell the product for the ROI it generates in your organization. 
4. The difference is your income.

Would you build the same product?

The role of testers

As a CTO, ten years ago, I said, "I pay my testers to find defects". This attitude has since changed.

I now think that when testers are finding defects, then someone, somewhere is doing something wrong. They shouldn't.

Instead, they should:

• establish a "test everything mindset" in the team
• ask the critical questions early on which allow the team to build a high quality product.
• work with the organization and users to guide efforts invested into establishing high quality.
• collaborate within the team to establish both technical and operative means for preventing low quality.
• exercise user empathy to help the team develop a product that doesn't just meet acceptance criteria, but even "feels right".
• take critical looks at the product to see which aspects weren't covered with Acceptance Criteria and identify improvement.

Plus a lot of other things.

And note how the idea of "finding defects" isn't even in there.