Cutting Corners - think about it ...

 I was literally cutting corners this weekend when doing some remodeling work, and that made me think ...

Cutting corners:

• is always a deliberate choice
• makes things look better to observers
• is what you don't want others to see
• doesn't require expertise
• provides a default solution when you see no alternative
• might be the most reasonable choice
• requires more work than not doing it
• is expensive to undo

So - try having a conversation: where are you cutting corners, why are you doing it - and do you know how much it costs? Which alternatives do you have? What might another person do different?

Microhabits - small action, big impact

 Let's talk about #microhabits - the small things that don't seem to make a difference at all in the short term, which are setting you on a long-term trajectory.

What are microhabits?

Microhabits are actions that take nearly no time and seem to have a very limited scope, and seem to not be worth mentioning, yet they set you on a compounding trajectory. Many years after adopting a microhabit, people adopting it are worlds apart from others around them.

Here are some examples of software development microhabits:

• Appropriately naming stuff
• Fixing typos
• Refactoring
• Making sure the code is easily testable
• Adding important unit tests
• Generally keeping code readable and workable
• Creating a working build at leasts a few times a day

No excuses

I often hear, "This was an emergency," or "This was just a demo," or "There was time pressure." These are supposedly justifications for not doing the things above. In the working world, there will always be some stress, deadline or emergency lurking behind the next corner. Everything else is cockaigne.

Here's the thing, though: Microhabits become "second nature" and it's more effort to break a habit than to pursue it, so we can't argue that pressure is a reason to do something slower, more complex and less routine than what we'd normally do.

People with good coding microhabits will pursue their habit and keep their code high quality regardless of circumstance. Simply because it's a habit. An important realization about habits: they'll never form if you constantly interrupt them - so consistency is key.

Form the right microhabits habits today!

Which actions, when done consistently over many years, will result in a codebase you'd love to work with? Adopt these, and keep doing them consistently.

And which would result in a codebase you'd loathe? Stop these, and avoid them consistently!

If you want to do a facilitated Retrospective on the topic, you can use this simple template:

TOP Structure - the Technology Domain

 Too often, organizations reduce the technical aspect of software development to coding and delivering features.

This, however, betrays a company which hasn't understood digital work yet, and it begs the question who, if anyone, is taking care of:

Technology is the pillar of software development that might be hidden in plain sight

Engineering

Are you engineering your software properly, or just churning out code? Are you looking only at the bit of work to be done, or how it fits into the bigger picture? Do you apply scientific principles for the discovery of smart, effective and efficient solutions? How do you ensure that your solutions aren't just makeshift, but will withstand the test of time?

Automation

What do you turn into code? Only the requirement, or also things that will help you do your work easier, with higher quality, and lower chance of failure? Do you invest into improving the automation of your quality assurance, build processes, your deployment pipeline, your configuration management - even your IDE? How many things that a machine could do is your company still doing by hand, and how much does that cost you over the year - including all of those "oops" moments?

Monitoring

Once you delivered something - how do you know that it works, it works correctly, is being used, is being used correctly, has no side effects, and is as valuable as you think it was? Do you make telemetry a standard of your applications, or do you have reasons for remaining ignorant about how your software performs in the real world?

All of the items above cost time and require skills. Are you planning sufficient capacity to do these things in your work, or are you accumulating technical debt at a meta level?

Think for a minute: How well does your team balance technological needs and opportunities with product and organizational requirements?

TOP Structure - the domain of Architecture

In software, there's a critical intersect between technology, that is - how we turn ideas into working software - and our organization - that is, who is part of development and how they interact.

Architecture is at the crossover point of Technology and Organization

This domain is Architecture, and it exists one way or another - if we don't manage it wisely, the outcome is haphazard architecture, most likely resulting in an inefficient organization delivering a complex, low value solutions in a long time and at a high cost.

Am I trying to advocate for a separate architecture team? No. Take a moment and think about Conway's Law: If we have the wrong organization, the consequence is the wrong architecture, the consequence is the wrong technology - and the consequence of that is a failing business.

Architecture is bi-directional. The right organization depends as much on technical choices as vice versa. We need a closed feedback loop between how we develop, and how we organize ourselves.
In many companies, the architectural feedback loop is utterly broken, hence they're doing with 50 people what could be done with 10.

One of the key organizational failures which lead to the need for "Scaling Agile" is that architecture is either disconnected from workplace reality, or not even considered to be important. By architecting both our organizational system and our technology to minimize handovers, communication chains and process complexity, many of the questions which cause managers to ponder the need for "Scaling Frameworks" are answered - without adding more roles, events or cadences.

This form of architecture doesn't happen in ivory towers, and it doesn't require fancy tools - it happens every day, in every team, and it either brings the organisation in a better direction, or a worse direction.

When was the last time you actively pondered how technical and organizational choices affect one another, and used that to make better choices in the other domain?

U-Curve Optimization doesn't apply to deployments!

Maybe you have seen this model as a suggestion how we should determine optimum batch size for deployments in software development? It's being propagated, among other places, on the official SAFe website - unfortunately, it sets people off on the wrong foot and suggests them to do the wrong thing. Hence, I'd like to correct this model -

In essence, it states is that "if you have high transaction costs for your deployments, you shouldn't deploy too often - wait for the point where the cost of delay is higher than the cost of a deployment." That makes sense, doesn't it?

The cause of Big Batches

Well - what's wrong with the model is the curve. Let's take a look at what it really looks like:

The difference

It's true that holding costs increase over time, but so do transaction costs. And they increase non-linearly. Anyone who has ever worked in IT will confirm that making a huge, massive change isn't faster, easier or cheaper than making a small change. 

The amount of effort in making a deployment is usually unrelated to the amount of new features part of the deployment - the effort is determined by the amount of quality control, governance and operational activity required to put a package into production. Again, experience tells us that bigger batches don't cause less effort for QC, documentation or operations. If anything, this effort is required less often, but bigger batches typically require more tests, more documentation and more operational activity each time - and the probability of Incidents rises astronomically, which we can't exclude from the cost of change if we're halfway honest.

Metaphorically, the U-Curve graph could be interpreted as, "If exercise is tiresome, exercise less often - then you won't get tired so often. The optimum amount of exercise is going to door to receive the pizza order, but rather order half a dozen pizzas at once if the trip to the door is too exhausting, and then just eat cold pizza for a few days." 

Turning back from metaphors to the world of software deployment: It's true that for some organizations, the cost of transaction exceeds the cost of holding. This means that the value produced but unavailable to users is lower than the cost of making that value available. And that means that the company is losing money while IT sits on undeployed, "finished" software. The solution, of course, can't be to wait even longer with not deploying, and losing even more money - even if that's what many IT departments do.

As shown in the model, the optimum batch size isn't achieved when the company is stuck between a rock and a hard place - finding the point where the amount of money lost by not deploying is so big that it's worth to spend a ton of money on making a deployment.

The mess

Let's look at some real world numbers from clients I have worked with. 

As I hinted, some companies have complex, cumbersome deployment processes that require dozens of people weeks of work, easily costing $50000+ for a single new version. It's obvious that due to the sheer amount of time and money involved, this process happens as rarely as possible. Usually, these companies celebrate it as a success when they're able to go from quarterly releases to semiannual releases. But what happens to the value of the software in the meantime? 

Just assuming that the software produced is worth the cost of production (because if it wasn't, why build it to begin with) - if the monthly cost of development is $100k, then a quarterly frequency means that the holding cost is already at $300k, and it goes up to over half a million for semiannual releases. 

Given that calculation, we should assume that the optimal deployment frequency is when the holding cost reaches $50k, which would be two deployments per month. That doesn't make sense, however: when 2 deployments costs $50k each per month, then 100% of the budget would flow into deployment - of nothing. 

Thus, the downward spiral begins: fewer deployments, more value lost, declining business case, pressure to deliver more, more defects, higher cost of failure, more governance, higher cost of deployments, fewer deployments ... race to the bottom!

The solution

So, how do we break free from this death spiral?

Simple: when you're playing a losing game, change the rules.

The mental model that deployments are costly and we should optimize our batch size to only deploy when the cost of deployment outweighs the holding cost is flawed. We are in that situation because we have the wrong processes to begin with. We can't keep these processes. We need to find processes that significantly reduce our deployment costs:

The cost of Continuous Deployment

Again, using real world data from a different client of mine: 

This development organization had a KPI on deployment costs, and they were constantly working on making deployments more reliable, easier and faster. 

Can you guess what their figures were? Given that I have anchored you at $50k before, you might think that they have optimized the process maybe to $5000 or $3000.
No! If you think so, you're off by so many orders of magnitudes that it's already funny. 

I attended one of their feedback events, where they reported that they had brought down the average deployment cost from $0.09 to $0.073. Yes - less than a nickel!

This company made over 1000 deployments per day, so they were spending $73 a day, or $1460 a month, on deployments. If we calculated the accumulated cost of deployments for the whole period, they were still spending over $5000 for three months' worth of software development. But the transaction cost for each single deployment is ridiculously low.

Tell me of anything in software where the holding cost is lower than 7 Cents - and then tell me why we are building that thing? Literally: 7 Cents is mere seconds of developer time!

With a Continuous Deployment process like this, anything that's worth enough for a developer to reach for their keyboard is worth deploying without delay!

And that's the key message why the U-Curve optimization model is flawed:

Anything worth developing is worth deploying immediately.

When the cost of a single deployment is so high that anything developed isn't worth deploying immediately, you need to improve your CI/CD processes, not figure out how big you should make that batch.

If your processes, architecture, infrastructure or practices don't permit for Continuous Deployment, the correct solution is to figure out which changes you need to make so that you can continuously deploy.

Test Automation Specialists - what a waste!

"Test Automation Engineers require someone who will write the test cases for them, and they can start automating right after the software is deployed onto the test environment."

It's 2022, and I just had this conversation yesterday. Seriously?

Overspecialization leads to Waste

If we work like that, we needlessly create bottlenecks, we test too late, our tests will be of low quality, and we will spend a lot of time on identifying, tracking and reworking defective deliveries. We produce a lot of work, but not quality.

In a well-functioning software organization, there can be no such role as a "test automation engineer" who is fed work by "test designers." 

Although it sounds "resource efficient," that approach generates massive amounts of waste in all categories of Lean Muda: 

Defects

When automated tests aren't written by developers, and only available once a software package is released, we end up with three kinds of defects:

1. Product defects, i.e. things the product should (or shouldn't) do, but that weren't correctly implemented, which require another software update.
2. Automation defects, i.e. things the test automation should (or shouldn't) do, but that weren't correctly implemented. Until clarified, these lead to significant communication overhead.
3. False negatives, i.e. inadequate automation not catching existing defects. This is significantly more likely when automation isn't written close to the code, and might omit dangerous cliffs in the product implementation.

Handover

Whenever actors in a process change, there's a handover. At the point of handover, we induce delay, lose information and risk the introduction of defects. This makes our process slow and error-prone. When test itself becomes a source of error, we should stop and think.

Motion

Handovers also mean that we're moving artifacts: test descriptions, build versions and defects - without need. We also need to coordinate this motion, which means we induce additional meetings into our process, which also slows us down and adds non-value added overhead. Motion also leads to inventory waste.

Inventory

As previously mentioned, un-automated tests will pile up as inventory, as will un-tested deliveries. When tests fail, the defect notifications are made available at a point in time when developers are working on other things. Hence, we are adding three queues to our process. Each queue contains work waiting to be done, and the combined length of all these queues is value delay for our customers.

Waiting

As mentioned in my article on CI/CD industry benchmarks, the time required between when a developer makes a change, and when a developer receives feedback on whether everything is correct shouldn't exceed a few minutes, preferably just a few seconds. If automation begins upon delivery, this period of time is impossible to warrant. The delivered product has to wait a significant amount of time until it can produce customer value, and developers will turn their attention elsewhere.

Overprocessing

Having different people automate tests and write productive code leads to "overprocessing," often called "gold-plating:" As mentioned in my article about the test pyramid, we should try to automate as close to the code as possible. When we're doing high-level tests which could instead be low-level tests, we make our test suite more complex, slower and less reliable than possible.Unit tests are extremely quick both to write and execute. Test automation engineers who automate black box/gray box tests will spend a significantly higher amount of work to write tests with poorer outcomes.

Overproduction

Since the amount of labor to create a product increment in server code and test automation are extremely hard to balance, the strategy of automating after the fact usually leads to developed product features piling up, waiting to be automated. As a consequence, the decision is often to release these without proper test automation coverage to flush the waiting queue. This builds up "technical debt" which deteriorates product sustainability.

Human potential

When developing software with multiple people, we act as a team. We can learn from each other, and we can align: Someone who can automate tests, can also write features. Someone who can write features, can also automate. As a team, we can align during planning "who does what," and fill our information gaps so that those doing the work have full information and don't need anyone else to fill in for them. Techniques like pair programming or code reviews ensure that even if one person makes a mistake, someone else will catch it before a defect even enters the system. It's a limiting belief to think that somehow people can't learn - we all can, if the conditions are favorable. Maybe not everyone will be the greatest developer, but we don't need to: we work within our ability, and grow our capacity a little every day.

To confine a test automation engineer to automating defined test specifications, and to confine coders to write code without understanding how to do that without defects - is an insult to our human intellect, and a very expensive one, at that:  All the above seven wastes cost a lot of money that we could save if we would just improve our collaboration, communication and skillset.

Lean Software Test Automation

Please, let me clarify. I don't believe that test automation expertise is a waste - the waste is actually hidden in the process, and how we use our talent.

A much leaner approach which avoids all of the above eight wastes is to educate and empower - use QA to help developers learn how to define the right tests, how to apply processes like Test-Driven Development and Behaviour-Driven Development, and guide them on their journey to deliver Zero Defect Quality. And in reverse, having developers upskill our test automation experts to help them deliver integrated pieces of value including working software.

The purpose of test automation in such an organization isn't to find defects - it's to maintain the high frequency of delivery of high quality products, and to serve as a living documentation for all behaviours of the product which is maintained as part of the development process without overhead.

Continuous Integration Benchmark Metrics

 While Continuous Integration should be a professional software development standard by now, many organizations struggle to set it up in a way that actually works properly.

I've created a small infographic based on data taken from the CircleCI blog - to provide an overview of the key metrics you may want to control and some figures on how the numbers should look like when benchmarked against industry performance:

The underlying data is from 2019, as I could not find data from 2021.

Key Metrics

First things first - if you're successfully validating your build on every single changed line of code and it just takes a few seconds to get feedback, tracking the individual steps would be overkill. The metrics described in this article are intended to help you locate improvement potential when you're not there yet.

Build Frequency

Build frequency is concerned with how often you integrate code from your local environment. That's important because the assumption that your local version of the code is actually correct and consistent with the work of the remaining team is just that - an assumption, which becomes less and less feasible as time passes.

By committing and creating a verified, valid build, you reset the timer on that assumption, thereby reducing the risk of future failure and rework.

A good rule of thumb is to build at least daily per team member - the elite would validate their changes every couple of minutes! If you're not doing all of the following, you may have serious issues:

• Commit isolated changes
• Commit small changes
• Validate the build on every single change instead of bulking up

Build Time

The amount of time it takes for a committed change until the pipeline has successfully completed - indicating that the build is valid and ready for deployment into production.

Some organizations go insanely fast, with the top projects averaging at 2 seconds from commit all the way into production - and it seems to work for them. I have no insights whether there's much testing in the process - but hey, if their Mean Time to Restore (MTTR) on productive failures is also just a couple minutes, they have little to lose.

Well, let's talk about normal organizations - if you can go from Commit to Pass in about 3 and a half minutes, you're in the median range: half the organizations will still outperform you, half won't.

If you take longer than 28 minutes, you definitely have to improve - 95% of organizations can do better!

Build Failure Rate 

The percentage of committed changes causing a failure.

The specific root cause of the failure could be anything - from build verification, compilation errors or test automation - no matter what, I'm amazed to learn that 30% of projects seem to have their engineering practice and IDE tooling so well under control that they don't even have that problem at all, and that's great to hear. 

Well, if that's a problem for you like 1/5th of the time, you'd still pass as average, but if a third or more of your changes are causing problems, you should look to improve quickly and drastically!

Pipeline Restoration Time

How long it takes to fix a problem in the pipeline.

Okay, failure happens. Not to everyone (see above), but to most. And when it does, you have failure demand - work only required because something failed. The top 10% organizations can recover from such a failure within 10 minutes or less, so they don't sweat much when something goes awry. If you can recover within the hour, you're still on average.

From there, we quickly get into a hugely spread distribution - the median moves between 3 hours and 18 hours, and the bottom 5% take multiple days. The massive variation between 3 and 18 hours is explained easily - if you can't fix it before EOB, there's an entire night between issue and resolution.

Nightly builds, which were a pretty decent practice just a decade ago, would immediately throw you at or below median - not working between 6pm and 8am would automatically botch you above 12 hours, which puts you at the bottom already.

First-time Fix Rate

Assuming you do have problems in the pipeline - which many don't even have, you occasionally need to provide a fix to return your pipeline to Green condition.

If you do CI well, your only potential problem should be your latest commit, and if you follow the rules on build frequency properly, the worst case scenario is reverting your change, and if you're not certain that your fix will work, that's the best thing you can do in order to return to a valid build state.

Half the organizations seem to have this under control, while the bottom quartile still seems to enjoy a little bit of tinkering - with fixes being ineffective or leading to additional failures. 

If that's you, you have homework to do.

Deployment Frequency

The proof of the pudding: How often you successfully put an update into production.

Although Deployment Frequency is clearly located outside the core CI process, if you can't reliably and frequently deploy, you might have issues you maybe shouldn't have.

If you want to be great, aim for moving from valid build to installed build many times a day. If you're content with average, once a day is probably still fine. When you can't get at least one deployment a week, your deployment process is definitely ranking on the bottom of the barrel and you have definite room for improvement.

There are many root causes for lower deployment frequency, though: technical issues, organizational issues or just plain process issues. Depending on what they are, you're looking at an entirely different solution space: for example, improving technically won't help as long as your problem is an approval orgy with 17 different comittees.

Conclusion

Continuous Integration is much more than having a pipeline.

Doing it well means:

1.  Integrating multiple times a day, preferably multiple times an hour
2. Having such high quality that you can be pretty confident that there are no failures in the process, 
3. And even when a failure happens, you don't break a sweat when having to fix it

And finally, your builds should always be in a deployable condition - and the deployment itself should be so safe and effortless that you can do it multiple times a day.

Thousands of companies world-wide can do that already. What's stopping you?

Definition of Done vs. Acceptance Criteria

Let's clear up a very common confusion: What's the difference between "Acceptance Criteria" and the Definition of Done, and which is which?

Sometimes, people ask "what is the Definition of Done for this User Story?"
And then, they proceed to document a feature's Acceptance Criteria like this:

1. Feature works
2. All tests passed
3. Documentation written

To make it short: in this example, AC's and DoD are backwards. Now, let's clarify.

Acceptance Criteria

A backlog item's Acceptance Criteria are stated from the perspective of the item's consumer: what defines, for them, that the item is successful?

I hesitate to say, "user acceptance", because sometimes, backlog items are not intended for the end user. If we ignore that case, though, the Acceptance Criteria could be written like simplified User Acceptance Tests: verifiable statements that can be answered with "yes" or "no", and tested by examining the product itself.

Good Acceptance Criteria would be, for example:

• I get informed about my current balance
• I get a warning when my balance is negative
• I get a notification when my balance has changed

As you see, Acceptance Criteria are things that the product does when the item is completed. They are not things that the development team does to the product.

Definition of Done

The Definition of Done is, as the term says, a definition, of what everyone in the development organization, the Product Owner, as well as stakeholders, should understand, when someone uses the term, "Done." 

The Definition of Done is a set of mandatory activities that apply to every single backlog item. It's a checklist-style itemization of things that the team does to the product in order to call a single backlog item "Done."

An example of a Definition of Done is,

• Code written
• All tests Passed
• Successful deployment to Test environment
• Mandatory documentation completed

When you have such a definition, it does not make sense to add these points again to every single backlog item.

But ... the DoD doesn't apply to this item!

Well, it's entirely feasible that the amount of effort to complete a specific DoD activity on an item is Zero. That means it's by default "Done", because all the work required is completed with zero work.

Another thing is when you specifically want to make an exception from your DoD for a specific item. Well, as long as it's an exception, you understand and accept the consequences, and everyone including the customers agrees to take this route, that's okay, because transparency is there.

Although I would caution that if it happens too often, you may want to check whether your DoD makes sense.

But ... it's a specific task for this item!

That's okay. Then add it that task to the item. It's still not an Acceptance Criterion, and it doesn't affect the applicability of the overall Definition of Done.

Discerning: AC or DoD?

	Acceptance Criteria	Definition of Done
Describes ...	What the product does	What the team does
Quality of ...	Product from user perspective	Work from process perspective
Applies to ...	Specifically this backlog item	Generally all backlog items
Met when ...	Verified by a test	Agreed inside team

The wrong professionals

Sometimes, I struggle with teams failing to understand the engineers' responsibility in quality: "I have asked the Product Owner whether we should apply Clean Code practices, and she said, she doesn't need it."

This is already not a conversation that should happen.

Here's a little metaphor I like to use:

 

And that has a number of consequences:

• Professionals do not negotiate the standards of professionalism with their customer. 
• Customers expect that the professional brings and adheres to professional standards, which is why they get hired. 
• Customers are not in a position to judge whether a professional standard is applicable in their context, and asking them to do so shifts responsibility to someone who can't bear it. That itself is un-professional.

So, what does that mean for your Agile team?

• Be as professional as you can, and continuously improve.
• Do not ask the customer to tell you what "professional" is.
• Instead, ask them whether your standards of professionalism satisfy their needs.
• You can't delegate the responsibility for the quality of your work to anyone else.
  The attempt is already un-professional.

The Code Review Department

 Let met tell you the story of a company that had problems with their code quality, and solved it the Corporate way ... well, ehum ... "solved."

Poor Code Quality

When I met that organization, their development process basically looked like this:

Developers took their specifications, completed their coding and passed the code for testing. Simple as that. 

After a major initiative failed, the Lessons Learned revealed out that the root cause for failure was the poor code quality, which basically made code un-maintainable, hard to read and difficult to fix defects without adding more defects elsewhere. 

A problem that would require rules and regulation to fix. So, it was decided that Code Reviews should be done. And - of course, people complained that code reviews took time, and if developers would do the Reviews, they would be even slower in getting their work done.

Stage 1: Introducing: the Code Review Department

What would be the Corporate way, except to introduce a new department of specialists dedicated to the specific task? So, they opened the Code Review Department. Let's call them CRD.

To keep Code Reviews as neutral and objective as possible, the Review Department was strictly separated from the others: physically, functionally and logically. They were not involved in the actual development process, and only reviewed the code that was actually delivered.

With the backing of senior management, the CRD adopted a very strict policy to enforce better code quality and defined their departmental KPI's, most notably:

• Remarks made per reviewer: How many review remarks each reviewer had found, per commit and per month. This allowed the CRD management to see which reviewers were most keen and could best spot problems.

Now, how could this possibly go wrong?

Stage 2: Code Quality improves

Since it's easy to objectively measure this data, the reviewers got busy and lots of review comments were made. Proudly, the CRD management presents statistics to IT leadership, including their own KPI's as well as the obvious metric they can present to measure the software delivered by the Software Development Department (SDD):

• Code Quality per developer: How many review remarks were made, per commit and month, per developer. This allowed the SDD to put a KPI on the quality of code provided by developers. And, conveniently, it would also justify the existence of the CRD.

With the blessings of IT leadership, SDD Management thus adopted the KPI.

So, things were going well ...

Stage 3: Code Review has a problem

Now, developers aren't dumb people. They aopted Linting and basically got their Review remarks down to Zero within in pretty short time. Now, the CRD should be happy, shouldn't they?

Turns out, they weren't. And here was the problem: The Reviews per reviewer metric tanked. Not only didn't reviewers suddenly fail their quota, CRD management figured out they probably weren't looking in the right places.

So what did the CRD reviewers, not being stupid, either, do? When they looked at code, they screened for patterns they considered problematic and introduced a new rule.

The numbers for Review remarks per reviewer rose again, and management was doubly happy: not only were the numbers of the CRD fine again, reviewers were continuously improving their own work.

Great! Things are even getting better!

Stage 4: Developers have a problem

Developers started to get frustrated. Their Lint rules were no longer working in getting them 100% passed reviews. What was worse, that they found whenever their Linting got updated, code was rejected again, and they needed to figure out the new rule to add to their Lint configuration. Not only did this process consume a lot of time, it distracted from actual development work.

Well, developers caught on that meeting the rules wasn't going to get them 100% reviews any more, so they began introducing honeypot infringements: They made obvious mistakes in their code so that reviewers would remark them, and they'd already have the fix in place right when the review came in.

Everyone happy: CRD met their KPI's and developers were no longer forced to constantly adopt new rules. Except ...

Stage 5: Management catches on

CRD reviewers were content, because they had plenty review comments again until the CRD management started to measure policy vialotations by type, and figured out that developers had stopped improving and were making beginner mistakes again. Of course, they reported their findings to higher management. And thus, a new KPI was born:

• Obvious mistakes per developer: How many obvious review infringements were made by team, with a target of Zero and published transparently throughout the SDD.

Well, again, developers aren't stupid people. So, obviously, they would meet their KPI. How?

You might have guessed it: they would hide their, ehum, "mistakes" in the code so they were no longer obvious, and then placed bets who could get most of them past Review without being caught.

Guess who won the game?

Stage 6: Code quality deteriorates

The CRD reviewers got stuck in a game of whack-a-mole with developers, who constantly improved their tricks of hiding more and more insidious coding errors, while updating their Linting rules right away when reviewers added a new rule.

Until that day when a developer hit the jackpot by splipping a Zero-Day exploit past Review. 

The CRD management no longer trusted their own Reviewers, so they added peer review reviews and another KPI:

• Issues slipped past Inspection: Reviews were now a staged process where after review by a Junior Reviewer, a Senior Reviewer would review again to figure out what the first reviewer had missed. Every Reviewer would get a Second-Review Score, and that score would need to be Zero. So, they started looking deeper.

You can guess where this is leading, right?

Stage 6: Code quality goes to hell

Now, with four-eye reviews and a whole set of KPI's, nothing could go wrong any more?

Code Reviewers were doing splendidly. They always had remarks and the department's numbers truly validated that a separate, neutral Code Review Department was absolutely essential. 

So the problem was fixed now.

Well, except one small fly in the ointment. Let's summarize the process from a development perspective:

1. When developers make mistakes, they are reprimanded for doing a poor job.
2. When developers make no mistakes, new rules are introduced, returning to (1).

Developers now felt like they were on a sinking ship. It was easier to simply continue making mistakes on existing rules than to adopt new rules. They came to accept that they couldn't meet their KPI anyways. 

Since they could no longer win, they stopped caring. Eventually, the review department was relabeled to complaints department, and nobody took their remarks seriously any more. Developers would now simply add buffer time to their estimates, and called it the "Review buffer".

By now, the CRD was firmly established, and they were also fighting a losing battle: try whatever they might, they got more and more overloaded, because truly necessary review remarks and more and more horrible code got more and more common. They needed to add staffing, and eventually outnumbered the developers.

The Code Review Department became the last bastion against poor code quality. A bulwark defying the storming seas of bad code. 

So, what's your take ... 

is a Code Review Department a good idea?

How would you do it?

Improving Code Reviews

A "code review" activity is part of many organizations' development process - and oftentimes, it sucks. It frustrates people, wastes their time and the value in improving quality is questionable at best. If you feel that's the case, here are some things you can try.

What's a Code Review?

"Code review" is a feedback process, fostering growth and learning. It should not be confused or conflated with a QA sign-off process. While finding problems in the code may be part of doing the review, that's not the key point.

So-called one-way "gate reviews" without feedback on defect-free code are a waste. A major portion of their value is missed. The best reviews won't merely help people learn where they messed up - they help people find new, better ways of doing things!

Now, let us explore five common antipatterns and what we could do about them.

Five Code Review Antipatterns and how to deal with them

Review Hierarchy

In many organizations, the Code Review process "puts people in their place": A more senior person reviews the code of more junior persons, and annotates everything that these did wrong. Yes - this sounds exactly like a teacher grading a student's term paper, and the psychological implications are very similar.

While this does indeed foster some form of learning, it creates an anhedonian mindset: the key objective of the coding developer is to avoid the pain of criticism and rework. There is little joy in a job well done. Deming's 12th point comes to mind.

Suggestion 1: Reverse the review process. Let the junior review the senior's code, and see what happens.

Suggestion 2: Do review round robins. Everyone gets to review everyone else's code.

Suggestion 3: Have an open conversation, "How do Code Reviews affect our view of each other's professionalism?"

Huge Chunk Reviews

I'll admit that I've been both on the giving and receiving end here: Committing huge chunks of code at once and sending thousands of lines of code for review in one big bulk, without any comments. And the review outcome was, "This is garbage, don't work like that!" Rightly so. Nobody in their right mind has time to ponder such a huge amount of changes in detail. The review feedback will take a long time and probably not consider all the important points - simply because there are too many.

Code Reviews shouldn't create a huge burden, and they should have a clear focus.

Suggestion 1: State the review objective: What would you like feedback on?

Suggestion 2: Send increments into Code Review that can be thoroghly reviewed in no more than 15 minutes.

Suggestion 3: Reduce feedback intervals. For example: no more than 2 days should pass between writing a line of code and getting it reviewed.

"LGTM" or whimsical feedback

Poor reviews start with the premise that "the only purpose of a review is to find problems." On the positive side of the spectrum, this leads to a lot of a standard "lgtm" (Looks Good To Me) annotations as code is simply waved forward. On the opposite side of the spectrum, some individuals feel an almost sadistic need to let others know that there are always problems, today stating "this is good", and tomorrow stating "this is bad."

Behind this antipattern is the "controller mindset" that someone in the organization believes that a review is intended to tell others, "you did this wrong, you did that wrong." 

You can improve this by moving away from checking the code towards positive reinforcement, creating virtuous learning cycles. 

Suggestion 1: Change the guiding question from, "What is wrong with this code?" towards, "What could I learn from this code?" 

Suggestion 2: Create Working Agreements how you want to deal with extreme ends of the review spectrum.

Suggestion 3: Collect the learnings from Code Reviews and look at them in the Retrospective.

Ping-pong or Ghosting

Small story: One of my teams had just fixed a production problem that was causing a revenue loss of roughly €15k per day. Someone from a different team did the code review, demanded some semantic fixes, these were made - next round: lather, rinse, repeat. After 2 weeks, the reviewer went on vacation without notice. The fix got stuck in the pipeline for 5 days without response. This funny little event cost the company over €250k - roughly three years' worth of developer salary!

Things like that happen because the expectations and priorities in the team aren't aligned with business objectives and also because of a phenomenon I call "ticket talk."

Suggestion 1: Make the Wait Time and Flowback caused by Code Reviews visible.

Suggestion 2: Create a Working Agreement to talk face-to-face as soon as there's flowback.

Suggestion 3: Replace Code Reviews with Pair Programming.

Preferences, emotions and opinions

Let's return to the "whimsical feedback" antipattern briefly. Many times, I see feedback over "use CamelCase instead of Snake Case", "use Tabs indentation instead of spaces" or whether a "brace should open behind the method name rather than in a new line". 

None of these make the product any better or worse. The debate over such matters can get quite heated, and potentially even escalate into a full-blown religious war. These are entirely up to personal preference, and as such, not worth a review annotation: They are red herrings. 

Suggestion 1: Formalize coding conventions and put them into your Lint / SCA config.

Suggestion 2: If you're really bothered, use a pre-commit hook to prevent checking in code that violates static code rules.

Suggestion 3: If you think a rule is missing or unproductive, bring it up in the Retrospective.

Alternative perspective

Code Reviews are just one way to improve coding within a team and/or organization. Mandatory code reviews - by default - create interrupts in the flow, reducing overall performance by a significant amount. Better options include:

• Code Review upon request
  (e.g., "I want to talk with you about this code")
• Code Dojos, where the entire team assembles to learn from one another.
  (SAFe's IP Iterations are great for dojos.)
• Pair programming - making the discussion part of the process.
  (Reviews should be obsolete if you do Pairing right)

Still, if your organization mandates code reviews, try to make the best from them.

Summary (tl;dr)

Code Review is more about fast feedback learning than about "catching errors".
A positive "what can I learn" attitude makes reviews much more enjoyable and beneficial than a negative "what did they do wrong" attitude.

When reviews expose pressing problems, don't just annotate them. Engage the discussion about "how can we work differently?"

The importance of testablility

In the article on Black Box Testing, we took a look at the testing nightmare caused by a product that was not designed for testing. This led to a followup point: "Testability", which is also a quality attribute of the ISO:25010. Let us examine a little closer what the value of testability for product development actually is.

What is even a "test"?

In science, every hypothesis needs to be falsifiable, i.e. it must be logically and practically possible to find counter-examples. 

Why are counter-examples so important? Let's use a real-world scenario, and start with a joke.

A sociologist, a physisicst and a mathematician ride on a train across Europe. As they cross the border into Switzerland, they see a black sheep. The sociologist exclaims, "Interesting. Swiss sheep are black!". The physicist corrects, "Almost. In Switzerland, there are black sheep." The mathematician shakes her head, "We only know that in Switzerland, there exists one sheep that is black on at least one side."

• The first statement is very easy to disprove: they just need to encounter a sheep that isn't black.
• The second statement is very hard to disprove: because even if the mathematician were right and another angle would reveal the sheep to not be fully black, the statement weren't automatically untrue, because there could be other sheep somewhere in Switzerland that are black.
• Finally, the third statement holds true, because the reverse claim ("There is no sheep that is black on one side in Switzerland") has already been disproven by evidence.

This leads to a few follow up questions we need to ask about test design.

Test Precision

Imagine that our test setup to verify the above statements looked like this:

1. Go to the travel agency.
2. Book a flight to Zurich.
3. Fly to Zurich.
4. Take a taxi to the countryside.
5. Get out at a meadow.
6. Walk to the nearest sheep.
7. Inspect the sheep's fur.
8. If the sheep's fur color is black, then Swiss sheep are black.

Aside from the fact that after running this test, you might be stranded in the Winter Alps at midnight wearing nothing but your pyjamas and you're a couple hundred Euros poorer, this test could go wrong in so many ways.

For example, your travel agency might be closed. Or, you could have insufficient funds to book a flight. You could have forgotten your passport and aren't allowed to exit the airport, you could go to a meadow that has cows and not sheep, and the sheep's fur inspection might yield fleas. Which of these has anything to do with whether Swiss sheep are black?

We see this very often in "black box tests" as well:

• it's very unclear what we're actually trying to test
• just because the test failed, that doesn't mean that the thing we wanted to know is untrue
• there's a huge overhead cost associated with validating our hypothesis
• we don't return to a "clean slate" after the test
• Success doesn't provide sufficient evidence to verify the test hypothesis.

Occam's Razor

In the 13th century, a monk by the name of Occam came up with what's known today as Occam's Razor, i.e., "entities should not be multiplied without necessity". Taking a look at our above test, that would mean that every test step that has nothing to do with sheep or fur color should be removed from the design.

The probability to run this test successfully increases by isolating the test object (sheep's fur) as far as possible, and eliminating all variability from the test that isn't directly associated to the hypothesis itself.

Verifyability

We verify a hypothesis by finding a counter-example to what's called the "alternate hypothesis" and assuming that if this one is untrue, then its logical opposite, called the "null hypothesis" is true. Unfortunately, this means we have to think in reverse.

In our example: To prove that all sheep are black, we have to sample all sheep. That's difficult. It's much easier to sample one non-black sheep, and thereby falsify that all sheep are black. If we fail to produce even one, then all sheep must be black.

Repeatability and Reproducibility

A proper test is a systematic, preferrably repeatable and reproducible, way of verifying our hypothesis. That means, it should be highly predictable in its outcome, and we should be able to test as often as we want.

Again, going back to our example, if we design our test like this:

1. Take a look at a Swiss sheep.
2. If it is white, then sheep are not black.
3. If sheep are not - not black, then sheep are black.

This is terrible test design, because of some obvious flaws in each step: 

1. The setup is an uncontrolled random sample. Since sheep are white or black, running this test on an unknown setup, means we haven't ruled out anything if we picked a black sheep.
2. The alternate hypothesis is incomplete: Should the sheep be brown, then it is also not white.
3. Assuming that step 2 didn't trigger, we would conclude that brown = black.

Since the "take a look at a Swiss sheep" is already part of the test design, each time we repeat this test, we get a different outcome, and we can't reproduce anything either, because if I run this test, my outcome will be different from yours.

Reproducibility

A repeatability problem occurs when the same setup can generate different results. In our example, "take a look at" could be fixed, assuming we take the mathematician's advice, by re-phrasing step 1 to: "Look at a Swiss sheep from all angles." This would lead everyone to reach the same conclusion.

We might also have to define what we call "white" and "black", or whether we would classify "brown" as a kind of white.

We increase reproducibility by being precise on how we examine the object under test, and which statements we want to make about our object under test.

Repeatability

Depending on what the purpose of our test is, we are doing well by removing the variation in our object under test. So, if our test objective is to prove or falsify that all sheep are black, we can set up a highly repeatable, highly reproducible test like this:

1. Get a white Swiss sheep.
2. Identify the color of the sheep.
3. If it is not black, then the statement that Swiss sheep are black is false.

This experiment setup is going to produce the same outcome for everyone conducting the test anywhere across the globe, at any point in time.

While there is a risk that we fail in step 1 (if we can't get hold of a Swiss sheep), we could substitute the test object with a picture of a Swiss sheep without affecting the validity of the test itself.

What is testability?

A good test setup has:

• a verifiable test hypothesis
• a well-defined test object
• a precise set of test instructions
• absolutely minimized test complexity
• high repeatability and reproducibility

When all these are given, we can say that we have good testability. The more compromises we need to make in any direction, the worse our testability gets. 

A product that has high testability allows us to formulate and verify any relevant test hypothesis with minimal effort.

A product with poor testability has a high difficulty associated with formulating or verifying a test hypothesis. This difficulty might translate into an increase of any or all of the following:

• complexity
• effort
• cost
• duration
• validity
• uncertainty

In conclusion

The more often you want to test a hypothesis, the more valuable high testability becomes.

With increasing change frequency, the need to re-verify a formerly true hypothesis also increases. 

Design your product from day 1 to be highly testable.

By the time you discover that a product's testability is unsustainably low, it's often extremely expensive to notch it up to the level where you need it.

What every Scrum Master needs to understand about Development

Not every Scrum Master is technical. Indeed, some of the best Scrum Masters I know have a sociology or psychology focus - so it's all fair and square to say that they're don't know how to develop code. This may (or may not) pose a problem depending on how advanced the team's engineering practices are: you can't get the team to deliver sustainable value when they don't know how to do that.

At a minimum, the Scrum Master should to realize whether the team is aware of what they're doing - and if they aren't, bring in a technical coach or senior developer who can teach the team the ropes. There are many nuances, and a non-technical person will not see everything. Once developers are on the right track, they can very well figure out where they need further help.


tl;dr: of all the categories below:
Absence of the practice causes delay, incurs cost, reduces stakeholder satisfaction and may kill the product (and in smaller organizations, the business!)

Here's what to look out for:

Automation

Phrased positively: If we do indeed automate everything that can be automated, we gain a much better understanding of what's actually needed to get something to work. Automated activity also frees developer's heads for doing new work rather than repeat activity. Everything automated can be tested, so we can test not only the software but also the meta-processes, bringing quality to another level. We create repeatability, reliability and confidence in the processes.
Plus, we eventually save time.

Dead giveaway? Team members have paper (or electronic) documents for standard activities they do all day.

Constant Refactoring

Refactoring should be a day-to-day activity as part of regular work. Every line of code should be refactored as soon as it becomes necessary.
Deferring refactoring will eventually put product development to a screeching halt, which can be fatal for smaller companies. I have seen products die entirely because one day, developers discovered that new features couldn't be integrated any more without breaking something else.

Dead giveaway? Developers complaining about code quality.

Unit Testing

Over the months/years, code will become more and more difficult to maintain unless there's a decent amount of unit tests. The time it takes to make even minor changes will explode in the absence of good unit tests: What might take seconds with unit test coverage could take weeks in untested code.

Dead giveaway? The "tests" folder in the repository doesn't get updates with every new version.

Continuous Integration

Problems in the codebase are often not visible until the latest version of the build (which must be subject to automated tests) has been validated. In highly complex environments, changes to one part of the code base may break other code, which will become visible only when those pieces play together. The risk of this happening increases with every change made. As developers do multiple things sequentially, the effort of locating the root cause of an issue becomes increasingly difficult to uncover with passing time, and every change ("fix") to an unstable codebase will amplify the problem.

Dead giveaway? Developers call things "Done", and there's no Working Build for it yet.

Code Smells

You don't need to be a developer to discover "code smells" which will eventually become a problem.

 Some of the most obvious code smells are:

• "Mountain Ranges": The code, rotated 90 degrees left, looks a whole lot like a mountain range. Such code is too complex to be feasibly testable
• "Copy+Paste": Code that looks eerily similar multiple times over or developers who outright say that they develop by copy+pasting existing code segments, then modifying them, produce code that eventually becomes unsustainable.
• "The Gigabyte Class": A class that is unusually large will eventually become a problem. As a rule of thumb, a class with more than 5000 Lines of Code is probably fishy. YMMV.
• "Treasure Hunt": When you see developers moving across myriads of classes all the time to understand what's going on, there's probably a code structure issue.

There's tons more of code smells, but these are the most common to see when developers don't know how to create Clean Code.

Dead giveaway? Developers don't understand code produced by other team members.

My discovery journey to FaaS

Moving from monolithic systems to autonomous services is more of a mindset challenge than a technical challenge, although depending on the architecture of a system, the latter can already be tremendous. In this article, I want to describe my own discovery journey which changed how I perceived software systems forever.

Introduction

I was working for a client on an Information Discovery project which gathered and evaluated data from a range of other platforms which were basically a Stovepipe Architecture, so we had brittle, incomplete data objects from a wide range of independent systems which all described the same business object. There were two goals: First, link the data into a common view - and second, highlight inconsistencies for data purification. So to say, it was a fledgling Master Data Management system.

Stage 1: The Big Monolith

Although I knew about services, I never utilized them to their full potential. In stage one, my architecture looked something like this:

The central application ran ETL processes, wrote the data to its own local database, then produced a static HTML report which was sent to users via mail.
The good news is that this monolithic application was already built on Clean Code principles, so although I had a monolith, working the code was easy and the solution flexible.

The solution was well received - indeed, too well. The HTML reports were sent to various managers, who in turn sent them to their employees, who in turn sent them to their colleagues who collaborated on the relevant cleanup tasks. This was both a data protection and consistency nightmare - I couldn't trace where the data went and who was working with which version. So I moved from an HTML report to server-side rendering within the application.

Stage 2: Large Components

My application became two - an ETL and a Rendering component, both built on the same database. I had solved the problem of inconsistent versions and access control:

But there was another issue: Not everyone needed the same data, and not everyone should see everything. There were different users with different needs. The Simplicity principle led me to client-side rendering: I would pass the data to the frontend, then let the user decide what they wanted to see.

Stage 3: The first REST API

I cut in a REST layer which would fetch the requested data from the backend based on URL parameters, then render these to the user on the browser:

This also solved another problem: I wouldn't need to re-create the report in the backend whenever I made changes to layout and arrangement. It enabled true Continuous Delivery - I could implement new representation-based user requests multiple times a day without having to do any changes to any data on the server. The cycle time was three to five relevant features - per day, and so the report grew.

Guess what happened next? People wanted to know what the actual data in the source system was. And that meant - there was more than one view.

Stage 4: Multiple Endpoints

Early attention to separable design and SOLID principles once again saved the day. I moved the Extraction Process out of the ETL application and into autonomous getter services: 

This made my local database asynchronous with the data users might see, but this was a price everyone was willing to pay - it was enough if the report was updated daily, as long as people had access to un-purified raw data in real time. Another side benefit was that I pulled in pagination - increasing both performance and separability of the ETL process as a whole.

Loose coupling allowed me not only to have separate endpoints for each Getter, it allowed me to completely decouple each getter from a central point: I could scale!

As the application grew in business impact, more data sources were added and additional use cases came in. (To keep the picture simple, I'm not going to add these additional sources - just imagine there's many more).

Stage 5: Separate Use Cases

This modification was almost free - when the requirement came, I did it on a whim without even realizing what I had done until I saw the outcome:

Different user groups had different clients for different purposes. Each of the client needed only a subset of data, and that data was a subset of the available data - I could scale clients without even modifying anything in the core architecture!

And finally, we had network security concerns, so we had to pull everything apart. This was a freebie from a software perspective, but not a freebie from an infrastructure perspective: I separated the repositories and deployment process.

Stage 6: Decentralization

In this final stage, it was impossible to still call it "one application": I ended up with a distributed system with multiple servers, each of them hosting only a small set of functions which were fully encapsulated and could be created, provisioned, updated and maintained separately. In effect, each of my functions was a service running independently of its underlying infrastructure:

And yes, this representation omits a few of the infrastructure troubles I had, most notably, availability and discovery. The good news is that once I had this pattern, adding new services became a matter of minutes - Copy+Paste the infrastructure provisioning and write the few lines of code that did what I wanted to achieve. 

Conclusion

As I like to say, "The monolith in people's heads dies last". Although I knew of microservices, I used to build them in a very monolithic way: Single source repository, single database and single deployment process. This monolithic approach kept infrastructure maintenance efforts low and reduced the complexity of maintaining system integrity.

At the same time, this monolithic approach made it difficult to separate concerns, pull in the necessary layers of abstraction to separate use cases, resulted in downtimes and a security nightmare: When you have access to the monolith, everything is fair game: Both availability and confidentiality are fairly easy to compromize. The FaaS approach I discovered for myself would allow me to maximize these two core goals of data security with no additional software complexity.

Thanks to tools like Gitlab, Puppet, Docker and Kubernetes, the infrastructure complexity of migrating from is manageable, and the benefits are worth it. And then, of course, there are clouds like AWS and Azure which make the FaaS transition almost effortless - when the code is ready.

Today, I would rather build 20 autonomous services with a handful of separate clients that I can maintain and update in seconds than a single monolith that causes me headaches on all levels, from infrastructure over testing all the way to fitness for user purpose.

This was my journey. 

Your mileage may vary.

How not to run an agile project

This is an example how an agile approach fails. It's a rather painful lesson of messing up and I write from my own perspective, without any intent to blame anyone - just as a note of caution for those who might be in a similar situation.

Introduction

I was working with a client who needed an important business critical system.
Their IT department was organized with traditional project management and ran on 1-2 months cycles. Because this product would be core of the business for years to come, I suggested to do this with internal staff, who would later maintain and enhance the product. The suggestion included putting up a cross-functional team with six people, one process manager, two analysts who should also produce test scenarios, two coders and a sysop.
I wanted to deliver incrementally using fast feedback cycles to both offer maximum value to the business quickly and close any potential gaps in understanding on the go.
And so it came to pass that I was to be the Project Manager of this team. Due to staffing reasons, I ended up with an external and an internal analyst as well as an external and an internal developer. No problem - or so I thought.

Mishap #1: Autonomy

The first battle my team fought was that of opinions about technology. Microservices were to be used, and the data was mostly complex and weakly structured. The team's choice fell on a document-oriented database, MongoDB. This choice was met with massive resistance from the developers outside the team who weren't familiar with this technology. There was over a month of debate whether this technology was appropriate, and nothing got done because we didn't even have a way to store data. We finally reached a point where senior management had to intervene. Even after that, no developer outside the team would bother to learn about this new technology. Strike #1 against the team, they didn't have the support of other developers.

Problem: The team wasn't empowered to make autonomous decisions to begin with.

Mishap #2: Continuous Delivery

Continuous delivery was the plan, and we had two-week iterations to begin with. Unfortunately, the company was a bit short on infrastructure, so we didn't get an environment for five months. I did let this pass, because it was a problem outside my sphere of control. In retrospect, it was what probably killed the project to begin with, because we couldn't deliver anything of value for a long time - and when we did, it looked staggeringly miniscule. As a team, we were now in the defense already.

Problem: No Continuous Delivery, no feedback learning.

Mishap #3: Mixed Responsibility

No bragging, just describing the role. My role included responsibilities from PO and SM functions, lead business analyst and tester, as well as those of a traditional Project Manager. As you can guess, this cocktail turned really sour. I couldn't be a neutral SM when I was pushing a technology choice, I couldn't be the PO focused on user needs while tracking tasks and making timelines and I couldn't be the team's confidate when pressing for results.

Problem: Scrum roles and traditional project roles don't blend - especially not in one person.

Mishap #4: Culture

From the beginning, we faced resistance from the remaining IT staff, cumulating in seemingly witty remarks such as, "Do developers pick their own work here?" or "Do you seriously want to tell me developers test their own code?" The culture of an agile team made the team an intruder, an outsider for the others. It attacked their very belief system. Combined with #2, we really didn't have anything to show, so the criticism grew louder with no results to counter them.

Problem: Simply working in the same room with people who don't understand what you're doing won't have the conversion effect you expect.

Mishap #5: Clean Code

Getting a small microservice out of the door shouldn't be much of an issue with five people and a couple of weeks on your time. Enter - lack of understanding of the "Simplicity" Principle. You won't be agile when people turn a simple SQL query into a battleship of complexity.
We lost months on un-maintainable legacy code because one developer took private code ownership, hard-coding tons of magic numbers and nesting over twenty loops - only to discover that stuff didn't work properly and nobody could figure out why.

Problem: Before trying to work in an agile manner, developers need at least a rudimentary understanding of software craftsmanship.

Mishap #6: Management

After the line manager saw that I wasn't properly commanding and controlling the team, he started pushing his superiors to "fix this problem". The result was that we ended up with two project managers - one to take care of the project and the line manager, who wouldn't take any responsibility except ensuring that people reported to him (mostly things which neither they nor he even comprehended) and met pre-appointed deadlines that were created in complete disregard of the time required to do the work.
This second manager caused developers to skip writing tests, checking in quick-and-dirty code, skipping on the agreed DoD to avoid punishment. Basically, this second manager invalidated our entire agile approach within weeks.

Problem: Team autonomy is a farce when someone outside the team says what by when. A PO who must accept such externally imposed constraints is a toothless tiger.

Mishap #7: Trust

The absolute elephant in the room was a complete absence of trust which I didn't address in a timely fashion. The more trust there was inside the team, the bigger the distrust of those outside became. The line manager became weary that people started to have opinions contradicting his own and could no longer be commanded and controlled so easily. Trust became an even rarer commodity as defined responsibilities weren't met, professional boundaries were violated and snippish comments about other people's actions proliferated.
I'm not going to exempt myself here, such a mindset is truly contagious and definitely not helpful.

Problem: As long as there are trust issues, doing work is irrelevant. When the trust can't be mended, it's better to end fast than drag on.

Summary

Yes. I messed up. I took a too large portion. I will avoid to take such an "omni responsibility" in the future, simply because I won't have either the time or energy to fix all the fatal problems. The Scrum value of "Focus" can't be maintained when meddling in too many issues at the same time. The biggest problem was trust. It is the root cause of all other problems - and letting trust issues go unchecked is a surefire recipe for disaster.

Lessons for the future

1. The team must be empowered to make decisions about their work autonomously. While input from the outside organization needs to be considered appropriately, the differing opinions from those not even doing the work should never paralyze the team.
2. Forget working in an agile fashion without the necessary support infrastructure. If you can't get it quickly, you might as well scrap the entire project. When it takes four months to get a build environment, your team is toast. For me personally, I will put an exit clause in future agile contracts that I will quit when my teams don't have build infrastructure within a week.
3. When one person thinks of a traditional project manager and another thinks of an agile Product Owner, conflict is pre-programmed. This conflict can't be solved by compromise. A clear role definition upfront goes a long way. As long as the client doesn't understand the difference, it doesn't make sense to start working. If the client is willing to learn, a solution can be found. Otherwise, continuous conflict is pre-programmed.
4. "Culture eats strategy for breakfast". I know this, yet it's so easy to forget in the heat of the battle. Taking on too many roles simultaneously makes it too easy to forget leaning back and see how culture is preparing its next meal. A pure coach can observe and have the right conversations at the right time. A PO/PM caught in delivery mode goes blind. I did. Against better knowledge, so I can better understand those who have the same problem.
5. You will only be as agile as your team, and "Continuous attention to technical excellence" is part of the agile principles for a reason. Working with developers who haven't kept up to date in years is another surefire way of producing an unsustainable product. In future PO roles, I will make sure there's a training budget and developers get proper training so that the best they can do is on top notch level.
6. Two managers doesn't work for the same reason as we don't have two Product owners or two POTUS. One must give the direction. There can't be two winning strategies that aren't the same. Next time someone tries pushes me to share a single management/PO role with someone else, I will resign. Immediately.
7. Trust is paramount. I thought trust would come over time as we would deliver. I was wrong. On so many levels. No Trust - no Done. In future endeavours, I will invest much further into trust building, and when I discover that's impossible, it's better to end fast than drag on.

"A bad system beats a good person every single time". I don't even claim to be a good person, but I do say that I learned a lot about spotting systems which are set up to fail.

And I hope this article is helpful for you, too.

Test Pyramid Explained - part 1

Let's take a deeper look at what the Test Pyramid is, and how it can help us achieve sustainable, high quality. In this section, we will take a look at the left part of the picture only, as understanding this portion is essential to making sense of the right side.

Yet another model of the "Test Pyramid" - there's more to it than meets the eye!

The five levels

Before we get into the "How", we will examine the "What" - the five different levels, starting from top to bottom. Why top-down? Because this is how the business looks at software.

Process Chain Tests

A process chain is a description of a user-centric feature (oftentimes, a user story), irrespective of where it is implemented. From a high level, a customer might be something like "I want my order shipped home."
Such a process chain may consist of a larger number of technical features realized across a bigger number of subsystems, some of them potentially not even software. In our example, the process chain might look like this:

1.  User presses "Purchase" (online shop)
2.  User makes payment (payment provider)
3.  Order gets sent to warehouse for picking (warehouse system)
4.  Order is picked (picker's device + warehouse system)
5.  Package is sent for shipment (logistics + logistics payment system)
6.  Package is shipped (logistics + logistics tracking system)
7.  Package arrives (logistics tracking system)
8.  Order is closed (online shop)

As we can see from this example, it's incredibly complex to test a process chain, as each system and activity has a chance to fail. The potential amount of failure scenarios are nearly infinite - regardless of how many we cover, there might still be another.

The good news is that if a process chain works, it's a guarantee that all subsystems and steps worked.
At the same time the bad news is that - if the process chain doesn't work, we may need to do a lot of trackbacking to discover where the failure was introduced into the system.

Regardless of how much we test elsewhere - it might just be a good idea to do at least one supervised process chain test before "going live" with a complex system. That is, if we can afford it. Many organizations might simply resort to monitoring a live system's process chain in a "friendly user pilot phase".

Duration
A process chain test might take anywhere from a few minutes to many weeks to complete. As a rule of thumb, lacking any further information, an hour to a day might be a solid guess for the execution time of such a test. This explains why we don't want hundreds of them.

System Tests

Slightly simper than process chain tests are the oftenplace common system tests: The system is considered an inseperable unit - oftentimes, a "black box".

A system test would be concerned with the activities and data transfers from the time data enters into one system until the sub-process within the system is closed. Resorting to our above example, a system test of the Online Shop might look like this:

1.  User presses "Purchase" (Webshop)
2.  User's order data is persisted as "Payment Pending" (Database)
3.  User is redirected to payment section (External Payment service)
4.  Payment is authorized (External Payment service)
5.  Payment authorization ID is persisted (Database)
6.  Order Status is set to "Payment Complete" (Database)
7.  User is redirected to "Thank you" page (Webshop)
8.  Order is forwareded to Warehouse system
9.  Warehouse System sends Order Acknowledged message
10.  Order Status is set to "In Process" (Database)

Here we see that system tests, despite having a much smaller scope than a process chain, are still nearly as difficult to test and stabilize. 

Oddly enough, many so-called "test factories" test on this level, creating complex automation scripts - oftentimes based on tools such as SeleniumIDE - which is seen as a feasible way to automate tests with little effort.

The downside of automating system tests is that a minor change in the test constellation will invalidate the test - in our example, if the "Thank You" is replaced with a modal stating "Your order has been completed.", we might have to scrap the entire test (depending on how poorly it has been written).

I have seen entire teams spending major portions of their time both figuring out why system tests failed - as well as keeping up with all those feature changes invalidating the tests.

Duration

System tests shouldn't take all too long, but 5-15 minutes for a single automated test case isn't unheard of. Fast system tests might finish in as little as ten seconds.

Integration Tests

Integration tests are inteded to check the I/O of a system's components, usually ignoring both the larger scope process chain and the lower level technical details. 

An integration test assumes that the preceding steps in the source system worked - the focus is on the system's entry and exit points, considering the internal logic as a black box.

In our webshop payment example, we might consider the following autonomous integration tests:

1. When a user presses "Purchase", all items from the basket are stored in the database (UI -> Backend)
2. When a user is forwarded to the Payment Website, the total purchase price is correctly transferred to the payment service (Backend -> payment system)
3. When a payment is successfully completed, the payment data is correctly stored (payment system -> Backend)
4. When an order is correctly paid, it is forwarded to the warehouse system (Backend -> warehouse system)
5. The Warehouse system's order acknowledge is correctly processed (warehouse system -> Backend)

Integration tests are much smaller than system tests, and the root cause of failure is much easier to isolate.
The biggest downside of integration tests is that they rely on the availability and response of the partner system. If a partner system happens to be unavailable for any reason, integration tests can not be run.
I've seen this break the back of one webshop's test suite who relied on a global payment provider's sandbox that failed to answer during business hours, because it was constantly bombarded by thousands of clients.

Duration
Integration tests don't do all that much, their downside is the response time of the two systems. Good integration tests shouldn't take more than maybe 50ms, while poor integration tests might take a few seconds.

A good way to speed up integration tests is by mocking slow or unreliable partner systems, which can also speed them up massively, but adds complexity to the component's test suite.

Feature & Contract Tests

This group simultaneously contains two types of testing, as these go hand in hand: Feature tests are the internal logic how a system processes data. Contract tests validate how the data is being passed into / exits from the system.

Here's an example of a feature test:

class BasketValidationResponseSpec extends Specification {
   def "information given to customer" (BasketPojo Basket, String message, Boolean status ) {

expect:
   Basket.statusMessage() === message
   Basket.checkState() === status

where:
   Basket | message | status
   [ Bread[1], Butter[1], Book[1] ] | "Valid" | true
   [] | "Empty basket" | false
   [Bread[199] ] | "Too much Bread" | false
   [Bread[1], Butter[199] ] | "Too much Butter" | false
  }
}

(please forgive my indentation, HTML indentation is a pain)

Feature tests don't rely on any external interfaces being available, making them both reliable and fast to execute. Unlike unit tests (below), they don't test the details of individual functional units, but focus on the interaction of multiple functional units.

Contract tests are the flip side of the coin here, as a feature test assumes that the data is both provided in the right way and is returned in a way that the interfaced component can correctly process. In an ever-changing software world, these assumption are often untrue - contracts help create some reliability here. I don't want to go into that topic too deeply, as contracts are an entire field on their own.

Duration
The good news is that good feature and contract tests execute in as little as 20ms, making them both incredibly fast and reliable.

Unit tests

The bread and butter of software development are unit tests. They test single individual functional units in isolation, and there should be no relevant system functionality that's not covered with a test that wouldn't explain how that functionality works.
The purpose of unit tests isn't as much to create user comprehensible test feedback, it's to ensure that the code is comprehensible and workable - even when refactored.

Unit tests will ensure your code is loosely coupled, that each method doesn't do too many things (ideal amount: one purpose per method), that involuntary design errors are quickly caught and many other things which help developers.

While well-designed Feature tests answer the question "Why" a piece of code exists, the unit test defines "How"it is implemented. Separating these two things often neither makes sense - the boundary may be fluid. The main difference is that a unit test never relies on anything external to the object under test, whereas a feature test might rely on full object instatiation.
Their main "downside" is that their lifetime is coupled to the functionality they test - whenever the functionality gets adjusted, the unit test either has to stay valid or needs to be modified.

Duration
Unit tests are extremely fast. There are even tools executing the unit tests of modified code in the background while the developer is still typing. The limiting factors here are pretty much CPU speed and RAM: executing an entire project's unit test suite shouldn't take more than a minute (excluding ramp-up time of the IDE), otherwise you're probably doing something wrong.



Given these definitions, let's do a brief...

Summary

Test Type	Duration	Accuracy	Durability
Process Chain	1h +	Very Low	Very Low
System	1-15min	Very Low	Very Low
Integration	50ms+	Low	Low
Feature&Contract	10-20ms	High	High
Unit	< 10ms	High	N/A

If you ask me if there's any sane reason to test on the higher levels of the pyramid - I'd answer: "It's too slow, too expensive and too unreliable." At the same time, there are reasons to test high in the pyramid, including: Coarse granularity business feasibility testing, lack of lower level automation and/or lack of developer skills.

In the -->next article of the series, I will explain the right side of the image - the testing metrics in more detail.