Measuring Agile Transition Progress

Many organisations ask, "How do we know if we're making progress?" - and, the answer is usually "To measure is to know". So, we need to have a measurement system which will help us determine success.
It does not matter whether we are looking to understand whether we are asking, for example, if our Scrum transition is making meaningful progress or whether our organization overall is improving. The underlying question "How do we know" and the need for a measurement system remains.

Ditch the Bullshit metrics

Before we dig into "How can we properly measure?", let us look at a few improper measurement systems. Many traditionally minded companies are trying to measure progress by looking at stuff like "Are we becoming better at following our Plans?" , "Do we produce more Lines of Code?"  - or even completely pointless metrics like "Do we complete more Story Points?".
Some still struggle with a Capacity Planning mindset and try to use "hourly estimates" to track progress.

Let us ignore for this post the widely varying specific reasons why these metrics are bullshit. There is a common underlying theme here: "All of these metrics are performance metrics." The problem you are trying to solve is not a performance problem, so why would you use performance metrics to solve a problem outside that domain? That's like measuring room temparature to figure out when the next bus arrives - bullshit.

Change your mindset for appropriate measurement

Regardless of whether you are looking at product development, agile transition or organization transformation - all of these also have a common underlying theme: You're not doing it for fun. You are trying to solve a problem. 

Therefore, the first step towards an appropriate progress metric: Admit that you actually have a problem - and acknowledge that you are trying to solve it. Without the honesty, "We have a problem and we want to know if we are solving it", you will never end up with an appropriate measurement system - because your measurement system should also acknowledge that you are dealing with the root cause of the problem.

Accept an undefined state

Traditional planning expects plans to be made up in advance, and then a report can be produced to state "Topic X = 20% progress". This, however, requires that you can predetermine the end point.

Unfortunately, when you are trying to solve a problem, that may not be so easy. Why do you still have the problem if you know exactly how to solve it? 

The first step is to acknowledge that there may be a total of 3 things you are not sufficiently aware of:

1 - Where do you stand (A)?

2 - Where do you want to go (B)?

3 - What is actually the best way from A to B?

A plan assumes that both A and B, as well as the path inbetween, is understood sufficiently well to define the path for transition. 

So, you must accept the idea that status measurement will not work out as well.

Define the Primary Issue

It's not as simple as stating "We did Waterfall, now we want to do Scrum" - that is not the problem you are trying to solve. Your problem is usually more deep-rooted, such as "Our customers are dissatisfied with the quality or delivery speed of our product" - otherwise, why would you change your current way of working?

Once you understand that you are not trying to go from Traditional Project Management to Agile Development, you will understand that you need to ask and answer completely different questions in order to even understand "Where do we stand?" - likewise, the question "Where do we want to go?" looks different.

For example, "We stand at ..." becomes "We are losing customers faster than we can gain new customers" - and automatically, the real question to answer is no longer "How can we increase self-organization?" - but: "What can developers do that would reduce our loss of customers?"

Consequently, the metric we'll be talking about is no longer driven by the agile framework of our choice, but by the primary problem we are trying to solve. 

Define transient success

Our agile transition is not defined "successful" when we are, for example, 97% compatible to Scrum based on the Nokia (Scrumbut) test - but when we have reached an organizational state where the Primary issue (and hopefully many others) is resolved.

We also need to understand that never does an organization only have one problem. Organizations are complex systems which are in themselves subjected to a chaotic system - the Market. By the time we have resolved one issue, it may already have changed - or, our understanding thereof may have changed.

Much rather than trying to make a better plan, we should simply accept that at any given time, we are not aiming to complete the plan, but trying to reach a more positive future state. When our understanding of the problem we have - or actually the problem itself - changes, then we should simply accept this. 

We can acknowledge that the future is shifting, the steps made in the past were useful - but that the next action in the present must be different from what we had assumed. 

Accept adaptive measurement

Every change that solved a problem yesterday can be considered yesterday's success. It was a real success, but today we need to solve a different problem, so we need a new way of discerning this success. In today's measurement system, yesterday's success may be insignificant or even counterproductive - so, we should no longer pursue yesterday's metrics.

Track meaningful problems

The most meaningful metric you can utilize is actually quite simple, and it's rather binary: "Are we actually solving our problems?"

We can elaborate this metric to a finer granularity by breaking down our Epic Problem.

For instance, "Poor quality" would refine into more intricate problems, such as: "Problems reported by Customer", measuring "How often and how long is the same problem being reported by customers before it gets resolved?" - "How many times did we miss an obvious software defect?" - "How long does it take us to catch a defect?"etc. etc. All of these metrics can then be easily drilled down and resolved, one by one.

Unfortunately, with these metrics in place, we most likely won't reach perfection, so our "target state" (e.g. Zero Defects@Customer, Zero Effort Testing, 100% Test Coverage) is more of a utopia than achievable. Therefore, tracking progress our "Epic Problem" becomes pointless.

Define how big you want the problem to be

The fact that our Epic Problem may never be 100% resolved should not stop us from defining precise targets for the definitely meaningful sub-problems.

For instance, we can determine: "Reduce defects in delivered product by 50% by end of year" - "Reduce acceptance test duration by 25% by next month without decreasing quality" - etc. These are specific, measurable, achievable, realistic and timebound: SMART measurement.

As seen from the two examples above, these metrics have no direct correlation with agility - likewise, they can become quite complicated to track. We can still use agile principles, practices and frameworks to improve these metrics. Afterwards, we can determine if the application of agility did help us reach these objectives.

Keep it simple and Lean

While some people feel the need for precision and metrical process control, we don't really need a complex, precise measuring system. Usually, we have a clear understanding of what is "good enough". We don't want to over-engineer. If someone says "We don't test enough", it may well suffice to say "Well - improve testing until we test enough". The person who said "It's not enough" may have a decent understanding of what "enough" means. 

You should only negotiate clear numerical goals if you presume disputing opinions.

Own your problems

The steps described above indicate that dealing with problems is very, very similar to the responsibility of a Product Owner - and that's no coincidence. You want somebody to take ownership. Someone who drives the resolution of the Epic Problems you are trying to solve - and also someone who drives the resolution of each workable sub-problem you are dealing with: You naturally end up with a Chief PO (Problem Owner) and Area PO's (Problem Owners).

Get a Problem Board

Probably the easiest way of dealing with your problems is to track them just like your team tracks the Stories on their Storyboard. You can see what's "toDo", what's "in Progress" - and what's "Done". This storyboard becomes the natural information radiator of your progress on your Agile journey.

Organize Problem Solving

If you are moving towards Scrum, it also makes sense to arrange an entire Scrum-like organization around the Problem Board. You can work with Plannings ("What problem will we tackle in this sprint?") - dailies ("What problem have we solved today - etc.") - hold Retros ("How did our problem solving go?") etc. etc.

Track Problem Solving

You are becoming "more agile" when you are getting faster and better at solving problems. Keeping a high level overview of how long it takes, on average, from the time you become aware of a problem until you no longer have this problem, is the best measure of how agile your organization is.

Summary

To measure the success of your agile transition, you must be willing to name and face your problems. The best indicator of success is how you deal with your problems. By institutionalizing problem solving, you both establish a transition measurement system and take specific action.

Do not look for "feel good" indicators. Accept problems as your primary metric. 

You will feel good when you have less problems.