Test Coverage

What you see above is a classical test report as you would expect it within a Waterfall project. Typically, test managers produce such statistics at frequent intervals to report the progress of testing.

As with any diagram, this one isn't worth much without explanation.

The blue bar is the amount of test steps conducted per day, the red bar is the amount of defects the software contained on each day.

Now, what you see is an amount of roughly 15000 test steps being run over the course of 2 weeks and roughly 800 defects discovered.

In large-scale Waterfall projects where I worked previously, this would have been a boatload of effort for a team of maybe 20 testers.

A job well done for the test manager, you can be sure QA would be praised.

What is this metric - really? 

It's the results of automated integration and acceptance testing for one of our products. What you see here is only my personal tests: Don't even get me started that I'm not a fulltime developer. All of the tests you see displayed here - accumulated - ran within less than 1 hour including defect reporting.
None of the defects discovered made it past the evening. All were fixed within the same business day!

The consequence of such activity?

We use Continuous Delivery, so we are able to deliver working software on a daily basis - and we haven't had a single critical fault in the system on a live installation.

Years ago, when I was working exclusively Waterfall, I couldn't have believed by myself that not only a single person could execute as many as 4000 test steps per day. I wouldn't have believed - and that is probably the more critical learning here:

A single change to a single component could wreak havoc in use cases that merely rely on interfaces and have no direct connection to the implementation of the changed component!
When a programmer might consider their work "Done" - without proper automated test coverage, these beasts are still lurking deep in the dark!
I wouldn't have run the tests if I knew that my code was bugged! I typically stay in development until I am confident that my code is defect-free! I mean, hey - I got unit tests to make sure I didn't do anything bad! (and that's already more than we often had in Waterfall projects)
In a Waterfall, I would have handed the topic off to test and waited for any potential bug reports.
And chances are, testers wouldn't have done a Regression test and wouldn't have discovered the issue before the change went live.

It's not that I became a crappy developer by doing Agile. On the contrary. All of my Waterfall experience is still there. I make fewer mistakes than ever. But mistakes still can happen. The difference is that they don't make it into production any more.

Lesson learned

If you're a Developer, you are well advised to automate tests for your components so that you know if you accidentally broke something. However, don't stop there. Unless you have integration test suites, you may not know when your work has detrimental impact on the system's overall functionality. Chances are if you are only automating functionality and not use cases, your software still works, but your users can't.

Hence, agilists say "Automate everything". (well, everything that makes sense) - it does pay off!