Are problems your problem?

Many organizations, and likewise the individuals working there, struggle with the notion of "Problems".
A "shoot-the-messenger" culture makes it impossible to name, much less claim ownership for, problems. To be able to still deal with reality, the very term "problem", inherently considered bad and a career-killer, gets sugar-coated or diluted. People start talking about "impediments", "issues" instead - and avoid doing that as long and much as possible.

Problems are not bad

Innately, a "problem" is just a "problem". It means that something does not work out as originally planned. This is not bad, it just means we have to change the plan - we might still reach the desired outcome.
The first step is to accept that "problem" is a neutral term, indicating "need for change" - neither good, nor bad.
 

Apples and ... Problems

Let's compare a problem to an apple. You can either get the value from the apple (eating it) or let it rot. Once you have the apple, you either take advantage of it, or it spoils.  That's totally up to you.
Treat your problems the same way: When you see one, try to get as much value out of it as possible. By ignoring it, your value will be Zero, but you will still be stuck with the cost of disposal.

Where do Problems come from? 

Can't we just make a better plan to avoid problems? Don't problems just reveal the incompetence of the planner? These are typical questions from a problem-averse company.
But we must accept that the world isn't simple. Problems arise not only because we aren't omniscient, but also because nobody's perfect and the world is changing. While some organizations expect their managers to be omniscient and everyone on their staff to be perfect, they still don't live in a bubble: The world around them is still full of imperfection and is changing at a rapid pace.
Because of this, we can't plan for everything - and even if we did, the plan would still fail. And that's a problem.

A problem is no problem

Many people consider it a flaw to admit having a problem. Nothing could be further from the truth: Being keen in observing discrepancies between plans and how they work out indicates a good analytic capability. The biggest concern is that in a complex system, a problem in one area may require adjustments in another area. Only by communicating that there is need for adaption and why this need arises will the entire system remain stable.

No problem is a problem

Denial of problems or being completely oblivious about problems in the surroundings is a typical response by those who have been conditioned by their environment. Sentences like, "In our organization, there are no problems" are common in companies where the assumption is that "sufficiently intelligent people can plan and anticipate everything". Not only are both denial and obliviousness problems - an environment where openness and integrity are not valued is another problem.
Usually, an environment where nobody sees a problem is the most unhealthy, because needed change is often procrastinated until too late.

Talk about problems

You can improve the effectiveness of yourself, your work - and your organization with a few simple steps:

1. Admit that problems exist. Call them what they are.
2. Don't shoot the Messenger. People who name your problems are not your enemy, they want to help you improve.
3. Live Continuous Improvement. Deal with problems before they grow out of proportion.

3 Ways to increase your Retro effectiveness

Inspect and Adapt is at the heart of agility: "Responding to change over following a plan". The Retrospective is Scrum's standard ceremony to discuss potential improvements. However, especially new teams struggle at making Retrospectives effective. Consequently, the ineffective Retro becomes an impediment towards agility. Here are a few tips for new Scrum teams to increase the effectiveness of their Retrospectives.

Get a good room

Since in a Retro, you expect to make a significant change towards a meaningfully positive change, do not let the room hinder the team from coming up with good ideas for change.

Focus

Every minute spent on sidetracks and gimmicks destroys organizational value. Everyone has the responsibility to keep focused during the Retrospective and the Scrum Master's facilitation of the Retro should support this process as naturally as possible.

Preparation

The best way to get a meaningful result out of the Retrospective is by being prepared. On the other hand, being completely unprepared is the best way to make any meeting for any reason whatever - completely ineffective.
Both good teams and Scrum Masters prepare the Retro in advance. Then, they follow through and turn this well-prepared Retro into meaningful and highly effective action.

Summary

Continuous Improvement, is the only way to reach high performance. The Retrospective is the standard mechanism of establishing Continuous Improvement.
On the downside, poor Retrospectives will result in inherently low agility.

A Scrum Master must frequently reflect, "How can we make our Retrospective more effective?" - and adjust their own behaviour accordingly.

Measuring Agile Transition Progress

Many organisations ask, "How do we know if we're making progress?" - and, the answer is usually "To measure is to know". So, we need to have a measurement system which will help us determine success.
It does not matter whether we are looking to understand whether we are asking, for example, if our Scrum transition is making meaningful progress or whether our organization overall is improving. The underlying question "How do we know" and the need for a measurement system remains.

Ditch the Bullshit metrics

Before we dig into "How can we properly measure?", let us look at a few improper measurement systems. Many traditionally minded companies are trying to measure progress by looking at stuff like "Are we becoming better at following our Plans?" , "Do we produce more Lines of Code?"  - or even completely pointless metrics like "Do we complete more Story Points?".
Some still struggle with a Capacity Planning mindset and try to use "hourly estimates" to track progress.

Let us ignore for this post the widely varying specific reasons why these metrics are bullshit. There is a common underlying theme here: "All of these metrics are performance metrics." The problem you are trying to solve is not a performance problem, so why would you use performance metrics to solve a problem outside that domain? That's like measuring room temparature to figure out when the next bus arrives - bullshit.

Change your mindset for appropriate measurement

Regardless of whether you are looking at product development, agile transition or organization transformation - all of these also have a common underlying theme: You're not doing it for fun. You are trying to solve a problem. 

Therefore, the first step towards an appropriate progress metric: Admit that you actually have a problem - and acknowledge that you are trying to solve it. Without the honesty, "We have a problem and we want to know if we are solving it", you will never end up with an appropriate measurement system - because your measurement system should also acknowledge that you are dealing with the root cause of the problem.

Accept an undefined state

Traditional planning expects plans to be made up in advance, and then a report can be produced to state "Topic X = 20% progress". This, however, requires that you can predetermine the end point.

Unfortunately, when you are trying to solve a problem, that may not be so easy. Why do you still have the problem if you know exactly how to solve it? 

The first step is to acknowledge that there may be a total of 3 things you are not sufficiently aware of:

1 - Where do you stand (A)?

2 - Where do you want to go (B)?

3 - What is actually the best way from A to B?

A plan assumes that both A and B, as well as the path inbetween, is understood sufficiently well to define the path for transition. 

So, you must accept the idea that status measurement will not work out as well.

Define the Primary Issue

It's not as simple as stating "We did Waterfall, now we want to do Scrum" - that is not the problem you are trying to solve. Your problem is usually more deep-rooted, such as "Our customers are dissatisfied with the quality or delivery speed of our product" - otherwise, why would you change your current way of working?

Once you understand that you are not trying to go from Traditional Project Management to Agile Development, you will understand that you need to ask and answer completely different questions in order to even understand "Where do we stand?" - likewise, the question "Where do we want to go?" looks different.

For example, "We stand at ..." becomes "We are losing customers faster than we can gain new customers" - and automatically, the real question to answer is no longer "How can we increase self-organization?" - but: "What can developers do that would reduce our loss of customers?"

Consequently, the metric we'll be talking about is no longer driven by the agile framework of our choice, but by the primary problem we are trying to solve. 

Define transient success

Our agile transition is not defined "successful" when we are, for example, 97% compatible to Scrum based on the Nokia (Scrumbut) test - but when we have reached an organizational state where the Primary issue (and hopefully many others) is resolved.

We also need to understand that never does an organization only have one problem. Organizations are complex systems which are in themselves subjected to a chaotic system - the Market. By the time we have resolved one issue, it may already have changed - or, our understanding thereof may have changed.

Much rather than trying to make a better plan, we should simply accept that at any given time, we are not aiming to complete the plan, but trying to reach a more positive future state. When our understanding of the problem we have - or actually the problem itself - changes, then we should simply accept this. 

We can acknowledge that the future is shifting, the steps made in the past were useful - but that the next action in the present must be different from what we had assumed. 

Accept adaptive measurement

Every change that solved a problem yesterday can be considered yesterday's success. It was a real success, but today we need to solve a different problem, so we need a new way of discerning this success. In today's measurement system, yesterday's success may be insignificant or even counterproductive - so, we should no longer pursue yesterday's metrics.

Track meaningful problems

The most meaningful metric you can utilize is actually quite simple, and it's rather binary: "Are we actually solving our problems?"

We can elaborate this metric to a finer granularity by breaking down our Epic Problem.

For instance, "Poor quality" would refine into more intricate problems, such as: "Problems reported by Customer", measuring "How often and how long is the same problem being reported by customers before it gets resolved?" - "How many times did we miss an obvious software defect?" - "How long does it take us to catch a defect?"etc. etc. All of these metrics can then be easily drilled down and resolved, one by one.

Unfortunately, with these metrics in place, we most likely won't reach perfection, so our "target state" (e.g. Zero Defects@Customer, Zero Effort Testing, 100% Test Coverage) is more of a utopia than achievable. Therefore, tracking progress our "Epic Problem" becomes pointless.

Define how big you want the problem to be

The fact that our Epic Problem may never be 100% resolved should not stop us from defining precise targets for the definitely meaningful sub-problems.

For instance, we can determine: "Reduce defects in delivered product by 50% by end of year" - "Reduce acceptance test duration by 25% by next month without decreasing quality" - etc. These are specific, measurable, achievable, realistic and timebound: SMART measurement.

As seen from the two examples above, these metrics have no direct correlation with agility - likewise, they can become quite complicated to track. We can still use agile principles, practices and frameworks to improve these metrics. Afterwards, we can determine if the application of agility did help us reach these objectives.

Keep it simple and Lean

While some people feel the need for precision and metrical process control, we don't really need a complex, precise measuring system. Usually, we have a clear understanding of what is "good enough". We don't want to over-engineer. If someone says "We don't test enough", it may well suffice to say "Well - improve testing until we test enough". The person who said "It's not enough" may have a decent understanding of what "enough" means. 

You should only negotiate clear numerical goals if you presume disputing opinions.

Own your problems

The steps described above indicate that dealing with problems is very, very similar to the responsibility of a Product Owner - and that's no coincidence. You want somebody to take ownership. Someone who drives the resolution of the Epic Problems you are trying to solve - and also someone who drives the resolution of each workable sub-problem you are dealing with: You naturally end up with a Chief PO (Problem Owner) and Area PO's (Problem Owners).

Get a Problem Board

Probably the easiest way of dealing with your problems is to track them just like your team tracks the Stories on their Storyboard. You can see what's "toDo", what's "in Progress" - and what's "Done". This storyboard becomes the natural information radiator of your progress on your Agile journey.

Organize Problem Solving

If you are moving towards Scrum, it also makes sense to arrange an entire Scrum-like organization around the Problem Board. You can work with Plannings ("What problem will we tackle in this sprint?") - dailies ("What problem have we solved today - etc.") - hold Retros ("How did our problem solving go?") etc. etc.

Track Problem Solving

You are becoming "more agile" when you are getting faster and better at solving problems. Keeping a high level overview of how long it takes, on average, from the time you become aware of a problem until you no longer have this problem, is the best measure of how agile your organization is.

Summary

To measure the success of your agile transition, you must be willing to name and face your problems. The best indicator of success is how you deal with your problems. By institutionalizing problem solving, you both establish a transition measurement system and take specific action.

Do not look for "feel good" indicators. Accept problems as your primary metric. 

You will feel good when you have less problems.

3 Pitfalls to avoid as a Product Owner

The Product Owner is the "single wringable neck" in Scrum. This implies that a huge burden is resting on their shoulders. However, we learned from Lean Management that "single points of failure" are a bad idea. So - how can the PO prevent being the SPoF?

1 - Micromanagement

Some product owners - and even Scrum teams - feel that all decisions about the Product, including features, UI and UX must be made by the Product Owner.
( Let us ignore for a second the fact that this is turning developers into mindless drones, because Software development is exclusively about decision making - the "implementation" is the easy part, finding out how to do it best is the hard - and fun - part. )

This implies that the Product Owner must be available whenever a decision is being made.
Also, that the Product Owner would need to understand the intricacies of UI, UX, SEO, SMO etc. (which would make the PO quite a jack-of-all-trades) - it means that the PO is considering the Product on a very base level.
The Product Owner should be more concerned with strategic propositions of the Product - such as which customer segment to target, what value proposition to offer - and how to turn a more or less clearly understood market need into a feasible backlog item.

2 - Being a designer

Some go as far as having the PO explicitly define all Acceptance Criteria and spoonfeed them to the team. There are two problems in this:

• This forces the Product Owner to spend a significant amount of time with intricate details of the "How" in implementation. 
• The product design is biased around the Product Owner's understanding of technology.

Let us consider a Site Landing, for example: "As a customer, I want to register so that I can use subscription services".
The Product Owner proceeds to design the Wireframe including all form fields and buttons, defines the events and hands that to the team. But that is bad for the product. Why?

Because the solution was pre-empted: Point in case - customers DON'T want to fill in registration forms. (just consider yourself: When was the last time you enjoyed that process?) Customers want to use services. Registration is the standard way of making subscription services available. But there may be different ways of identifying subscribers, such as - referring to an existing Google / Facebook accounts, etc. - you might even want to use facial or fingerprint recognition. That's easier for the user and serves the same business purpose. And users don't end up with the umpteenth password to remember.

3 - Being a Scrum Master

While the Product Owner is responsible for the success of the product, they are not responsible for enabling the team to work un-impedited. Scrum clearly discriminates the Product Owner and Scrum Master roles, for multiple reasons.

One of these reasons is, of course, that the Product Owner wants maximum value in the shortest possible time - while the Scrum Master might have to shoo back the Product Owner on decisions affecting technology that might compromize sustainability of the Product. The most common decision (un-enlightened) Product Owners make is to cut short the testing in order to speed up delivery. In these cases, a Scrum Master must intervene.

If the Product Owner perceives the urge to go about educating others about Scrum and resolving team impediments, this means that the Scrum Master isn't doing their job. No accusation to the Scrum Master - maybe that's an organizational issue. But if the Product Owner does the job of the Scrum Master, then the team remains in dysfunction: Not only does the burden on the PO increase, the key impediment does not get resolved.
It is better for the Product Owner to step back, sit with the team and Scrum Master - and find a solution for this issue than for the PO to sprint into action.

Summary

Being a good Product Owner may - occasionally - imply doing something that is definitely outside the sphere of your core responsibility, i.e. envisioning and packaging a great product. However, if you see that you are spending a significant amount of time with stuff that should be done by others, ask yourself "What am I doing here?" - give it to the team, and refocus.

You will only succeed as a Product Owner if you do what a PO should do best - not when you do what others should be able to do better.

3 characteristics of outstanding Product Owners

The success of your product depends to a large extent on your Product Owner. If the PO directs the product in the right direction, your success chance will be maximized. If not ... your product may well fail.

How do you decide the future of your product?

Here are three characteristics of outstanding Product Owners:

1 - Obsessed with customers

Customers are not merely those who pay for what you produced. They will use the product - they know what annoys and what pleases them. They are not just consumers. They have opinions, ideas.

Every Product Owner must understand their customers, how they act, what they think, what they want.

Good Product Owners have a good understanding of who their customers are and how they use the Product - and what they will pay for.

Great Product Owners drive this to obsession, constantly looking for new and better ways to understand their customers better, to get closer to their customers.
They become the personification of "the customer" when discussing the next Product Increment - and they become the avatar of "the Product" in their interactions with the Customer.

Outstanding Product Owners are emotionally attached to the Customer and the Product. They feel joy when something is done well, but they also suffer when customers suffer, they feel pain when the product is inadequate - and they can't "not take it personal".

2 - Caring for numbers

Numbers are not everything, but numbers are important. You can't be a good Product Owner if you only groom and discuss user stories. You must correlate them to the needs of the customer and the business.

Every Product Owner must understand the Product Vision, communicate it clearly and base Releases, Epics, Stories and Features on this vision. They must always be able to draw lines forward and backward between these items and be ready to restrict everything that is not in line with the product vision.

Good Product Owners are not only able to map stories. They can relate them to business figures. They understand which Epic, story or feature delivers how much business value. They can clearly discriminate between "gold plating" and Minimum Viable Product. They will deliver the highest value first and cut discard items which do not have a good bottom line.

Great Product Owners will go one level beyond that: They do not only attach a business figure to features, they relate Performance Metrics, such as Clicks, Conversion Rate and hard cash earnings. They do not only predict which feature is most valuable - they build a product that lets them measure, so that they can inspect and adapt their product to maximize value and customer satisfaction.

Outstanding Product Owners transcend all of that - they have the data and are ready to produce accurate analysis at a fingertip. Additionally, they understand the large scale implications of a change: They do not focus only on the product, but on the ecosystem in which the product exists. This includes being aware of potential on short-term gains that might decrease future sustainability - for example the phenomenon called "Mudflation" in MMO's.

3 - Disregarding agendas

Of course, the Product Owner works in your organization. But the loyalty of the Product Owner lies with the Product, not with a specific portion of the organization. They own the product and the product directs them - not management!

Every Product Owner must be in charge of their product. If you're looking for someone to whom you can spoonfeed what to do when, you're not looking for a Product Owner - you're looking for a Project Manager. That's OK, but don't call it Scrum, don't claim to be Agile - and don't wonder why you're wasting lots of money on a failed product: It was a management decision.

Good Product Owners take ownership, just like the title of the role proclaims. This means that they will do what, based on their understanding, is the right thing to do. If management wants the Product Owner to do a certain thing with the product, they will provide clear information on why they want that - and must be willing to accept a "No". A good Product Owner will tell management if an idea is stupid.

Great Product Owners care more for the product than for themselves. They will do the right thing, even if it is a personal disadvantage. They can not be threatened or scared by backroom politics or titles. Neither will they be swayed by personal gain - you can not bribe them with pocket money or promotion. If a suggestion is bad for the product, it will be rejected. Likewise, just because a suggestion is good, it will find it's place in the Product Backlog based on product value, not on someone's title or rank.

Outstanding Product Owners are beyond your company. They choose to dedicate a portion of their life to the product - if you want something from them for a political reason, be prepared for rejection. If you have a suggestion that would destroy their product, be prepared for fire. If you want to gut the product for short term gain, you have to accept that they will make the product successful without you.

Summary

The decision making process of an outstanding Product Owner is driven by customers, numbers and value. You can't own an outstanding Product Owner - and they own the Product.

Before you hire a Product Owner, decide if you really want a Product Owner. Your choices are still to hire a Project Manager, an Analyst or a Requirements Engineer. None of these are Product Owners.

An outstanding Product Owner is not a person to whom you simply delegate work in the development process.

3 things you won't get from me as a Product Owner

The Product Owner is the "single wringable neck" in Scrum. Their understanding of the product and the vision combined with their ability to decide determines success or failure of a product development team.

So, it's only fair that the Product Owner should be able to provide every information about what's going on? No.

Here are 3 things which you will not get from me when I'm a Product Owner.

Assignments

If you are a team member, do not ask me to give you an assignment.

I don't assign stuff because I couldn't. I could. Easily. But I don't. Here is why:

Even before the Sprint, I determine what matters most to the Product. During Sprint Planning, we mutually agree on the team's priority. Your top priority is visible on the Scrum board. It's the topmost item that is not in the "DONE" column.

If you are uncertain which task to pick, you have team members to help you out. Again, during Sprint Planning, the team has agreed on the necessary work items required to get the Stories or Features properly done. I let you, as the team, break it down because you know your trade. And I expect that when you do that, you know what's most important and what is the next step from where you currently stand.

If you don't understand the context of something you work on, I'll gladly discuss with you. If you don't know if something is necessary in order to deliver the value, I'm also open. But I won't tell you what to do - or how to do it.

If you, as a team member, don't know what to do next, I see a serious impediment in team communication. That's something to discuss with the Scrum Master or cover in the next Retro. But the Product Owner is really the wrong person here.

Gantt Charts and Reports

If you are a manager, don't ask me for a Gantt Chart or a % progress report on your backlog items.

I don't produce these because I couldn't. I could. Easily. But I don't. Here is why:

I decide what gets done and when the team engages. I can't tell you when it's guaranteed to be done. You wouldn't want a trash product - you want quality.
Percent Progress Reports are a distraction. You already know from Windows Installer, "The last 1% may take longer than the first 99%". It takes as long as it takes.

I can tell you how big your chunk is and where it sits in my backlog. I can give you an estimate if we'll tackle it in the next sprint, within this quarter - or not.

If you're not satisfied with my answer, we can re-negotiate priority and scope, thereby getting your stuff through the door faster, but we can't negotiate quality or delivery date. Sorry.

Performance Appraisals

If you are Human Resources, don't ask me for individual performance appraisals of developers.

I don't give you an assessment because I couldn't. I could give you my opinion. Easily. But I don't. Here is why:

First, because it's just my opinion. It's tainted by how much I have interacted with that person. It does not reflect total contribution.

Second, is the team producing significant value or not?
If not, that's my problem. If they do, they are doing a good job and then that's your assessment.

Third, let me ask: Why do you even need individual appraisals? Is it "rationalization"?
Let me be witty here: You don't ask your dentist which of your teeth works best, so that you can pull those out that contribute least. You'll want to keep all of them, unless you want to reduce your diet to pulp and soup.


A team is a team. Teams have very complex mechanics. Teams work as teams. If they don't, ask the Scrum Master what they are doing. If they do, then each person has their part.

If you want any information about the team, ask the team. They know best.

Summary

As Product Owner, I will give you the most valuable product in the shortest possible time. I am not the Team Leader or a Project Manager, despite the fact that I have significant experience in those things.
We will never get anywhere as long as I act as if I were the brightest bulb in the lamp.
I trust the self-organization of highly competent individuals to do what is right.
If that is not happening, that's something to improve upon - not to undermine.

No changes to the Sprint?

A very common question that is echoed by many Scrum teams: "What if we discover during the Sprint that a story will not help the user?"

The problem manifests in different facets, such as: A story was improperly groomed, new information became available which invalidated old information, a hypothesis about the product was disproved, etc.

Let's ignore for a minute the fact that "if we had known better, we would not have accepted this story into the sprint". We didn't and so we did. It's there. Now what?

The easy (and wrong) answer

Following the strict Scrum rules, the question is easy to answer: Continue the Sprint, deliver, use the Retro to elaborate what went wrong and do it better next time.

That's obviously wrong
We're not in business to "do Scrum". We're in business to deliver working, useful software. If we're not doing that, we're losing our reputation, credibility and - consequently, our customers - in turn: our jobs.

The correct answer

Common sense dictates the correct answer: Do what is right. Now, what is right? We deliver working, useful software.

Rather than referring to Scrum, we need to refer to the Agile Manifesto in answering this question:

Let's start with the often-ignored first sentence: "We are uncovering better ways of developing software by doing it and helping others do it." Is it "better" to build something useless? Obviously not. Being agile indicates that we're doing the best we can and from there, we look for even better ways. By simply doing what we're told, we neither do the first, nor the second.

Now, let's screen the Agile Values:

People and Interactions over Processes and Tools - you don't want to follow "the Scrum process" when a face-to-face communication can solve a problem that only exists because of a process.

Customer Collaboration over Contract Negotiation - when your "sprint contract" says to build and deliver X, yet X doesn't make sense to the customer/user, collaborate to find the Y which does.

Responding to Change over Following a Plan - when new information reveals that the Sprint Plan is no longer valid, respond to this change.

Scrum or Agile?

Scrum is an agile framework. The Agile Manifesto is the basis of Scrum. Without it, Scrum is a hollow, lifeless shell that will fail to deliver value. It would merely be a cargo cult, a "scrum-but" of the worst kind.
Scrum thrives by living the Agile Manifesto.

Refer back to the Agile Manifesto before clobbering someone with the Scrum Guide or Scrum Primer.

Summary

When you discover mid-sprint that your initial sprint plan (or, even a part thereof) does not make sense - grab your team, including the Product Owner and find a solution.
Never follow an invalid sprint plan: Change the plan, but do it in an agile fashion!

SysOps and Development: An Antipattern

Developers develop, Sysops operate. Obvious.
Well, that's classical thinking.

The consequence is that problems do not get resolved on time: Users suffer, value is lost and the reputation of the IT department is tarnished.

Obvious problems often don't get resolved in a timely manner - the loser is: everyone!

What you see above is real data taken from an organization where some Scrum teams are responsible for creating features and another team is responsible for operating the platform.

Just taking this example, the biggest current issue is an encoding problem - something that happens in the "real world", but not in a controlled development environment: Continuous Integration was successful, a feature hit the market - and boom!

Separate Responsibilities

In the example above, developers are busy churning out new features while SysOps are busy with fire-fighting.
Taken from the chart, it's been 3 days since the problem started - 3 days where SysOps are constantly stressed while developers are busy with stacking more features on top of existing problems.

XP proclaims "A Red Master is developers' Priority 1" --- well, whose priority 1 should a Red Production be? SysOps or everyone?
In a classic Kanban system, when the production line goes down, it should stop the entire machinery until the issue is resolved. This also holds true for agile teams: If anywhere in the organization, there is a problem, stop. Fix.
Don't go on producing more problems.

Perverse incentives

Developers are appreciated for delivering new features, while SysOps are appreciated for keeping the platform stable. Consequently, Devs prefer to work on new stuff rather than fixing existing stuff. Unfortunately, the organization as a whole suffers. Not rowing in the same direction can't yield any better results.

The very fact that people have different incentives here implies is a problem with autonomy and self-organization!
You must align the incentives of everyone working on the product, not only "development team".

Definition of "Done"

Regarding developers higher for delivering a new feature than making an old feature stable yields the problem above: "Just add a new story into the Product Backlog and we'll deliver it with the next sprint." - while this is a classic approach to Agile Software development "There are no bugs, missing Acceptance Criteria are new stories" and Scrum's "The Sprint is Closed towards modification".
This is a severe misunderstanding: Can you actually consider a story "Done" when it causes problems to the customer?

When you see that there's an operative problem with a new feature, it means that in your retrospective, you should put your DoD under scrutiny.

DevOps: A solution

As the agile proverb goes,  "You build it, you run it."
As long as developers do not feel the operational pain, they are not interested in removing it. Likewise, as long as SysOps are unable to fix the root cause of a problem in source code, they become apathic towards code-related problems.
Moving both the operative problem as well as the solution space into the team's sphere of responsibility, developers will take an interest not only in the code they write and how it meets the acceptance criteria, but also how the code actually performs in the real world.

Organizationally, this can be done as simple as moving a SysOp into the development team and then holding the team accountable for events on the productive platform.
The Developers become Dev-Ops who will come up with innovative solutions to detect, analyse and prevent problems. A DevOps can solve issues in a fashion that would be completely unthinkable for a SysOp who is confronted with a "build ready" black box software.




Side note
The above screenshot was taken from Medullar, towards which I, personally, am completely biased. I am one of the core developers of this easy to use, open source, cloud-based, universal monitoring and problem solving solution which was designed to require minimum levels of intrusion and resource capacity while offering maximum flexibility.
Medullar allows you not only to detect and analyse problems in any kind of environment. It provides an API and UI to remedy them in real time. It even comes bundled with it's own test framework, so you don't need anything else to improve your operative performance.

What's better: Scrum, Kanban or XP?

The question "Which agile framework is the best?" is often raised by people who are fairly new to the world of agile frameworks. In short: I can't answer this question without asking a counter-question: "Best for what?"

 

 

When your only tool is a hammer, every problem becomes a nail!

Let me analyze the problem in regards to the image above.

Scrum: keep things in place

Scrum is primarily known for the sprint cycle process and the ceremonies: Grooming, Refinement, Planning, Daily Standup, Review and Retrospective.
Scrum discerns only the roles Product Owner, Scrum Master and Developer. You need a Product and Sprint Backlog consisting of Stories, in the latter broken down into Tasks.
You'll need to define suitable Working Agreements, such as a Definition of Ready and Definition of Done with your team, then use Inspect+Adapt on these rules to optimize your process continuously.

Scrum is quite rigorous. If you do not have any of these things, you end up with a disdained "Scrum-But". If you do something beyond, you usually also end up breaking your Scrum.

Scrum is a wonderful framework for creating structure, clarity and order. Additionally, Scrum reduces volatility, enhancing predictability and confidence.

Kanban: Create flow

Kanban is often confused with the Kanban board, the most notable artifact. Making work visible, creating a smooth flow of work as well as identifying and removing bottlenecks are all key factors of successful Kanban.

Kanban is very flexible. You can start with Kanban wherever your organization is, without interference to structure or process. From there, you start optimizing.

Kanban is a wonderful framework for making impediments visible and optimizing delivery.

XP: Professional Craftsmanship

Extreme Programming is mostly concerned with Engineering Practices. Techniques such as Pairing, Test Driven Development, Continuous Integration and Refactoring will lift your developers to new levels of excellence.
Design concepts such as Whole Team approach, Collective Code Ownership, Clean Code, Emergent Design will broaden your team's horizon and capacity.
Lean concepts such as sustainable pace, Metaphors, Small batches, Standards , Automation and Continuous Improvement reduce waste and risk.

XP is process-agnostic. You can pick individual practices from XP that solve specific needs in your team, although taking the batch is advised for best results.

Summary

None of these agile frameworks is "better". Each of them has a specific area of use. As such, the frameworks become "better for solving a specific problem". Therefore, you need to define your problem first before answering the question "Which framework is best?" - a hammer is better for fixing a nail, but not for a screw. Of course, you can cut a board with a screwdriver, but a saw would suit your need better.

Take some time, define your specific problem, then work from there.
Here are some rules of thumb:

• Solve organizational or process issues with Scrum.
• When you can't seem to get anything "done", try Kanban.
• When your perceive quality or architectural issues, apply XP.

These frameworks are not exclusive. You can mix them - and high performance teams combine them. Scrumban is famous, for instance.

Disclaimer: None of these frameworks will actually make you successful in the long term without an Agile Mindset. If you have issues with Agile Values or Agile Principles, your only fix is returning to the Agile Manifesto - regardless of which framework you're using!

Why Test Driven Development?

TDD is sometimes proclaimed as a means for early discovery of all defects. Sometimes, people who are new to Software Craftsmanship practices wonder what tools they should use for TDD.

What TDD is not about

TDD is not so much about what tool you are using. TDD is tool-agnostic. The best way to do TDD is to use the same programming language. If you're looking for tools, you look in the wrong direction.
It's not about having a high test coverage. It's not about minimizing quality risks. In fact, it's not a Quality Assurance technique at all. If you use TDD for finding bugs, you're definitely on the wrong track.

What TDD is about

The key to successful TDD is to produce code in a fashion that each component is fully inspectable down to the most granular level, so that:

* You have a good software design

* You fully understand what each component, method and API does

* You minimize overhead of future reverse-engineering

* You lay a basis for easy changes, so that now you don't need to waste time on thinking about possible future changes

* If you need to change something fundamentally, you know where to best do it

* If something doesn't work as intended, you can isolate the root cause quickly

Summary

The main reason to employ TDD is not if you have "lots of defects". Chances are that you will not only not reduce the defect count with TDD on a poor quality legacy application - but that you will invest a lot of effort into increasing coverage without getting any measurable benefit to the user (customer) at all.

The best reasons to employ TDD are: when developers complain that they can not make changes to the software easily or they complain that finding the root cause of defects takes forever.

TDD is architecture+design topic.
The first minute you start doing TDD, each new line of code written, you will be working towards a better software design. It definitely pays off in the long term. 
Just don't look at the release defect metric, because you will not see the benefit there.

The five key ingredients for successful software development

All you need is good code that meets the requirement and you're set? That is short-sighted!

You need to keep a keen eye not only on the following key processes and artefacts to be successful in the long term.

Working Software (Code)

That's obvious. If you don't have a product, you have nothing.
This not only includes what the customer sees, but also what is going on behind the scenes. The statement, "A kitchen where someone cooks is always messy" might apply to the developer's office, but it must not apply to the code.

Smelly code costs real money and binds capacity that would be better focused towards innovation and progress.

Test Suite

To understand whether your product works correctly and does the right thing, you need a proper test suite. I radically proclaim "If your test isn't at least as clean as your productive code, don't bother".
A common (anti-)pattern I observe is that organizations de-prioritize testing in order to meet certain goals. Or, they actually put an emphasis on testing, but approach testing wrongly. The consequence? Flaky or unreliable tests that cause more trouble than they solve.

The money invested into testing is misspent if your tests are not well-managed, clean and comprehensible. However, abolishing your test suite equals abolishing your investment into sustainability. Don't ever reach this point.

From day 1 of a new product, invest into a proper test suite. Treat your test suite as the Holy Grail of long-term sustainability.

If you have existing software with that, go seek that Holy Grail - today!

Working Build

To actually reach Continuous Integration - or beyond, you need a smooth, swift build process. You should be able to modify your build as easily as you can modify your product's features.

I see many teams and companies who under-emphasize their build process. A sad fact is that they all run into build problems sooner or later. Some can't manage their dependencies any more, some don't understand why their build is unstable - and the build becomes a nightmare for every developer!

Like you manage your code and the test suite, you must manage your build.This includes the build management system, the build artifacts as well as the build infrastructure.

Environment

A good product is not impeded by the environment, which includes technical infrastructure as well as corresponding tools and integrated components.

An overly complex or under-performing environment are equally detrimental to the success of your product.

For example, the wild growth of turnkey service solutions is the best way to lose control of what is actually going on. It is quite easy, for instance, to include a tracking or ad service. It is, however, incredibly hard to debug your software if such an external service is causing a malfunction in your product.

Another example, if you are using a myriad of virtual machines without having a proper hardware management will cause tremendous trouble when trying to find out what went wrong where, and why. The time lost to a poorly managed environment can quickly exceed the time you are actually spending on the product you're building.

 

Runtime

"Works on my machine" - isn't it a running gag?

How do you control that the product also works in production? Of course, you need monitoring and feedback.

But technical monitoring doesn't cut it.  Have you ever taken a detailed look into your log files? Whenever I engage with a new client, I take the liberty of scanning the productive logs for stack traces and the other obvious stuff ("Exception", "ERROR", "CRITICAL", "FATAL"). Doing that only takes a minute with grep and I usually find a myriad of stuff for the next retro.

Sometimes, the software doesn't display any obvious sign of malfunction for your (Dev-)Ops while thousands of users around the world have no clue why they can't reach their intended goal.

To understand and control your runtime properly is a tremendously important - yet oftentimes underestimated - success factor.

Conclusion

You must take care of five critical ingredients:

Code, Test, Build, Environment and Runtime.

Give sufficient attention to all of these will drastically reduce your risk of failure.
Ignore one of them and you will find yourself struggling to keep customer satisfaction and progress up.

Failing to plan is planning to fail!

The Agile Manifesto boldly states, "We value ... Responding to change over Following a plan". I can't reiterate often enough that these are not mutually exclusive conditions.

I remember working in a project which coined the idiom "headless chicken mode" to describe the activities resulting from new feature requests: Everyone got busy, running into different directions - but nothing substantial resulted.

While panic and blind actions are response to change, they certainly are not the best. What the Agile Manifesto is concerned about is not plans in themselves - but that new information tends to make old plans obsolete!

Even seasoned developers occasionally fall into the habit of not planning, time and again! Throw out a (big) task that is fun and people will jump head in - without considering the long term objective! The result? When the window of opportunity closes, we have great ideas and fascinating code, but no marketable product!

Here are some things which should definitely be planned:

What do we hope to accomplish?

If you are uncertain as to what you will have accomplished when you are done, you will have no idea of how to know whether you have met the goal yet. It is completely legitimate to change your objective once new information uncovers that another course makes more sense.

It is not okay to be vague or unclear about your future accomplishment.

If you can not state your objective in one sentence - plan better!

In which order do we want to deliver what?

Agile approach suggests going MVP rather than perfectionism. Often, it takes a whole lot of thought to discover what is actually the minimum viable product and when we can deliver it.

The faster you can deliver something, the faster you will know whether you properly understood the need. Set short term goals that help you meet a long term goal. Don't go for that Big Hairy Audacious Goal without delivering value time and again in between!

If you have an ambitious plan that could take two months or more, don't set out unless you can break that plan into at least 3 steps that you can put into delivery order. Plan better!

How do we Inspect and Adapt?

When an idea is new, it tends to be coarse, ambiguous and with a lot of room for change. This uncertainty is no problem - unless we fail to manage it.
We can not remove uncertainty by drawing a straight line through uncharted territory - but what we can do: Take small steps, then carefully plan the next step.
Your Inspect+Adapt plan also needs to include how you will know whether you have made progress in the right direction - or not. Using direct customer interaction is as valid as some Big Data solution like Google Analytics or Omniture. Just make sure you choose the one that suits your needs.

If you can't describe your I+A strategy in one sentences, you have failed already! Plan better!

What do we need to succeed?

You should be clear what you will need to be successful. The Bible states, For which of you, desiring to build a tower, does not first sit down and count the cost, whether he has enough to complete it? Otherwise, when he has laid a foundation and is not able to finish, all who see it begin to mock him, saying, ‘This man began to build and was not able to finish'

You don't need to sign up to Christianity to understand the wisdom in these words. It is okay to miscalculate. It's also okay to miss something. But it's definitely not okay to not even bother figuring out whether you even stand a sporting chance!

If you can't name the top 3 success contributors and your strategy for obtaining them: don't start! Plan better!

Summary

This list of planning factors is by far not exhaustive, there are many more things to consider. In general, agile plans do not imply Gantt charts or big documents - but a clear, concise strategy for what you will do in order to succeed.

Your two most valuable plan items that need full clarity are:

• The "big plan", a.k.a. the Product Vision - where you want to be
• The next step

At any time you have uncertainty about those two things, stop what you're doing and re-plan!

Being a valuable Product Owner

In the Product Owner training, we learn that the Product Owner is in charge of the Backlog, and therefore controls "What" is being done "When". With this, the Product Owner leads Product Value - and: success.

Those are the basics.

Many times, you see "Fake Product Owners" in corporations. Retrained team leaders, project managers or even analysts.

They are handed requirements or, even worse, detailed specifications by other departments and are then expected to utilize the team to turn these into Working Software.

Let me be blunt here: If you have a good team, you don't need them. Because they can't do the job they've been set to.

It is not a Product Owner's job to play "Chinese Whispers" between business and the development team.

Understand the Customer

If you want to be a good Product owner, you must understand your customer. Please ask yourself a few questions about the product:

• Do you know any real customer and what they think?
• Do you really know what your customers are and will be doing with the product?
• Do you have facts, rather than opinions, regarding what the customer pays money for?
• Can you correlate features with relevant business figures? (more on this later)

A witty side reminder to PO's working in corporations: Unless you are working on an in-house product which is exclusively being utilized by staff of your company, your customers are not internal stakeholders, such as your Division Head, your Marketing Department, your Legal Department or your CxO.

If you do not have any access to real users and their needs, you are, at best, a proxy. Most likely, you'll be doing a political project where success is not determined by hard cash - which means, you won't be able to contribute anything relevant.

 

Determine Priority

Your team may be doing a tremendous job, but they are not omnipotent. They can't do everything at the same time, and they probably also shouldn't.

You probably heard the proverb, "Get your priorities straight".

Let me tell you: It's wrong. You don't have priorities. You have exactly one priority. Your first priority is "to satisfy the customer through early and continuous delivery of valuable software".

To do this successfully, you must be able to determine what satisfies the customer most, so that you can set your team to work on the one thing which provides the most value to the product.

Weeding the Backlog

You have a backlog rather than a requirement list mainly for one reason: You already know that the amount of requested work is higher than the amount of available resources.

Don't even bother with anything that can not be delivered in the near future, also don't even bother with anything that won't satisfy your customers.

 

You must say "No" to anything which you do not consider sufficiently valuable. Don't beat around the bush. Let people know "No means no." Please refer to "Weeding the Backlog" to see one approach to obtain a thinner backlog.

If you can not say "No" to anything put on your desk, you will be doing a miserable job. 

Learning how to say "No" so that people will not be personally offended with you is an art, however.

Slicing to the Bone

I love hate it when people approach me with "world hunger type" feature requests. Yeah, I understand that you want a widget that has thingummywat to do oomlah. But Rome wasn't built in a day.

Stakeholders love to push Epic work into your team, because they feel that they get something significant. Let me tell you what: Don't bother unless they can cut it down.

It isn't sufficient to merely throw a feature over the fence, groom it, develop it. That's not your job. Your job is to extract the Minimum Viable Product from each and every feature!

If you can't fit it into a single sprint, you are probably not working on an MVP.

To give you a specific example of what I mean: I can produce a skeleton CRM System with a single team in one sprint. But it won't do much. It will just tell me (and my customers) what must be done next to bring the most value.

Cut out everything that's not vital. Maybe that means cutting out the entire backend - maybe it means cutting out the entire frontend. But you must get useful, working software fast.

Keep the time!

Based on your customer, the priority and with minimal slices, be prepared to deliver what the customer wants, when the customer wants - with high quality. The only thing you must never promise is that the customer gets the entire scope they might have imagines.

Never, ever, postpone deadlines to stuff in more features.

I recently saw from a DevBlog for a new video game: "We could have made the skill wheel in 3D, but it would have cost too much development time." - throw out the chaff, never get yourself into a mess where you have to stand empty-handed before your customer because your scope didn't fit the time.

Your customers will be forgiving on the scope, but not on not having anything on the agreed date. Deliver. On time. 

Increment afterwards as long as there is value to harvest.

Summary

You are responsible to deliver a satisfying, valuable product in short, frequent intervals. If you do this well, the customers will be praising your product and your team - and you.

If you don't do this well, you'll quickly become everyone's punching bag.

Being a good Product Owner requires a solid understanding of your product, your customers and then making tough choices and "having balls". 

You won't please everyone. If your managers or internal stakeholders are an impediment to success, you need to bulldozer your way. 

If you can't or don't dare do that, Product Owner may not be the right job for you.

Done, Done-Done, Done Done-Done?

The "Definition of Done" is a fundamental artifact in Scrum.
It allows the team to decide when they can safely consider their work "Done".

However, many times, team fall into the trap of "Done-Done".
The root cause is usually external dependencies, such as being unable to test software ("Story Done" vs. "Test Done") or a separate Operations team ("Go-Live Done").

At scale, matters quickly worsen: Sprint-Done doesn't mean that there is a Working Release, much less that it has been deployed to Production.

As a first step, the solution is usually to come up with separate definitions of Done.
When describing these, please note that I mean: "First Step" of improvement, not something to be proud of. Having half a dozen of Definitions of Done is a smell, an indicator that something is very, very wrong!

Here are some definitions I have already encountered in the past:

Story Done

The basic activities required to say "We're done with development". Basically, this typically contains fulfilling all Acceptance Criteria, Code Review, Pull Request approved. Ideally, the Acceptance Tests are fully automated as well, although that one is already a compromise in many large organisations.

Sprint Done

The basic activities required to say "We're done with development". Basically, this includes that the Product Owner has accepted all delivered Stories and having merged all Stories into a Green Master.

Test Done

After developers have built "Working Software", it must be deployed and running on an integrated test environment where you can test the interaction with other components and services as well as with real people.
Some parts of the tests that may need to be done include Integration Testing, Performance Testing, Exploratory Testing etc. Just refer to the Test Quadrants to get some ideas of what may be needed.

Deployment Done

After tests are passed, the software must be Up and Running on a Live System. Alternatively, if you are shipping Vendor Software, it means you have a Shippable Product that has gone out to the customer. 

Release Done

 A Release is "Done" when no more activities related to the release are oustanding. Beyond deploying Working Software, it may include automated monitoring and reporting, looking at first figures on the Live Server etc.

The illusion of "Done"

At worst, you end up with a state of Done-Done-Done-Done-Done-"Not Done-Back To Step 1", where faulty stories need to be fixed weeks after they were marked "Done".
As a consequence, old work starts to flow back into current sprints, reducing predictability, velocity and morale.

Oftentimes, companies take this venue because either:

• There is too much pressure to get something "Done" 

If this is the case, management must be educated that they are sacrificing a high amount of sustainability for an insignificant increase of speed.
Listing the "hidden factory" work that must be done by the team outside of "Story Done" and figuring that into the total cost of delivery may help.

• They don't know how to do it differently

 From a strategic level, this is easy to fix. Operatively, it requires a lot of organizational change.

 

Minimize "Definitions of Done"

Team Definition of Done

A "Story" should not be "Done", followed up with additional work to get the "Sprint Done". The first step is to eliminate this gap.
"Story Done" should be attained when there is no more work that the team can do in the sense that from the team's perspective, there is Working Software by the time Story is set to "Done". A story is never "done" unless it is already integrated into the Master and ready to deploy to Production.
Provide a single Definition of Done for the team that includes everything the team can do. Accept no compromise, no intermediary steps.
Other teams can do this, so your team can do it, too!

Organizational Definition of Done

Never take the route of splitting into stuff like "Testing Done", "Release Done", "Deployment Done".
When setting out, catalog all "undone work" which the team does not do that happen before actually bringing the software into production into a single "Organizational DoD" (ODoD).
After creating a catalog, start out by eliminating the obvious waste.
Then comes the tricky part.
Every item on the ODoD should be considered an impediment for story delivery: Anything "urgent" is always blocked by the ODoD.
Depending on your organization, it may take years to move all items from the ODoD towards the team. This should not discourage you from undertaking the journey: Every reduction to the ODoD increases the organization's flexibility and risk of failure in case of emergency.

The Vision of "Done"

There is a single Definition of Done for each team.
The teams act upon this definition and not anything else.

The team can complete all activities necessary to reach Done.
No other party must become active for any item to become Done.

"Done" means "Done".
There is no work left to do when the team moves the card to "Done".

We know it's over.
The risk of failure is already eliminated, so there will be no residual risk of having to undo.

Weeding out the backlog

When companies transition to Scrum, or any other Agile methodology, they usually start with the problem of arranging all the identified "undone work". Where do we start?

In the worst case, huge amounts of "overdue" items are all considered "Priority 1". In this situation, the Product Owner must clearly indicate where the team should start.

Other organizations seem to suffer from a kind of collection addiction, keeping a backlog of hundreds of items that won't ever get done.

Here are some tips for maintaining a nice, tidy, workable backlog.

Intially setting up a backlog

Before even bothering to produce detailed user stories or doing task breakdown, the Product Owner should simply draft out the basic areas of the Product. 

Post-It notes with maybe one or two words will do the trick. Then, draft the matrix below on a board and place the sticky notes:

The backlog item placement matrix

Put your backlog items on this matrix!

Impact

When weeding out based on impact, you must identify your customers and stakeholders. For each item you identified, you must ask "Who will suffer if this is not implemented?" 

If the impact of not implementing the item is low or not even measurable, the item is certainly not important.

Urgency

Next, you weed out based on urgency. A good initial question to ask when chaffing out for initial urgency is: "If we do not deliver within 1 month, will there be significant damage to our business case?" - if the answer is "no", the item has low urgency.

Unclear cases

If you're unsure, feel free to put the item on the dividing line between low/high.

When that region clutters, you can benchmark by using the top item on the dividing line. Ask yourself for each item: "Does this have higher impact / urgency?" to make a decision quickly.

Scrap

Ideally, you will end up with a good number of items in the "Scrap" section. Congratulations, they are off the table! You do not need to waste any further minute with these topics.

Defer

For anything in the section "Defer": put those items on the bottom of your backlog and leave them there. Do not waste time now grooming them, because you will not work with them in the near future.

Why?

For anything in the section "Why?":

If you would invest a lot of effort for little impact, move the item to the "Scrap" section.

When you're just starting a new product, it's probably a bad idea to have anything from this in your backlog, because you want to get your product going - NOW!

When you're working on an existing product, you may want to look for "quick wins" in this section. Anything that will give you a credibility boost at low cost should be considered.

As long as the item still has a positive Return on Investment, you should do it, but only if you can squeeze it into the schedule.

Only groom items which will bring your product ahead with minimal effort. 

Move the others into the backlog until further notice.

Do Now

Should you have significantly more than five items here or dispute over priority, you need to weed out further.

The items remaining in this section are your high level backlog ("Epic items").

These items need to be made workable, such as by using INVEST. Break them down to feature level and start with the Story Breakdown.

Ordered weeding

If you feel uncertain about being too lax with weeding and up with a cluttered "Do Now" section", you should use "Magic Numbering" on the cards twice.

During the first round, simply arrange the cards by impact. After finishing, arrange them into buckets from 1 to 5 with ascending impact and note the impact on the cards. Unless you have less than 6 items, each number should have at least 1 item.

During the second round, arrange the cards by urgency from "low" to "high" in ascending order from left to right.

Then, use the noted numbers of impact to move the cards up (high impact) or down (low impact). Make sure their horizontal position remains unchanged. when arranging them on the vertical axis.

Afterwards, draw the lines - not in the middle, but between the "3" and "4" rows and columns, so purposefully create a bias towards "High".

Wrapping it up

You may flinch when you see that "lots of valuable stuff has gone off the list", but if you did the arrangement correctly,  only "even more valuable stuff will get done".

Your initial backlog should be very small and tidy at this point, easily workable for further breakdown.

Likewise, you can immediately and effectively quench discussion about any other topic with the simple argument "It won't get done now, anyway".

Recursion

You can apply the same process with the story / feature breakdown of each item. When moving into the first sprint of the team, it's actually a good idea to not overload the team with information. Weed the Ready ("groomed, broken down, workable") backlog to no more than five workable items for the first sprint.

Sprint planning

After the sprint, you will learn about the team's capacity. The amount of items you need to have Ready for the next Sprint should be roughly equal to twice the team's velocity. 

Once an Epic ("high level") item is fully groomed and there are not enough Ready items in the backlog, you take the next Epic and break it down. Start with the "Do Now" items and pull from "Defer" or the "Quick Wins", depending on what currently makes more sense.

Iterating

Impact and Urgency are subject to change. For instance, in May, the annual fiscal report is not urgent - in December, that's an entirely different story! Or, you have a customer threatening with a lawsuit - that may make an otherwise insignificant topic very urgent.

Whenever an item is added to / removed from the Epic Backlog, priorities may change. Between each sprint, you should bother taking a couple of minutes to do an iteration of weeding to make sure you are properly set up for the next sprint. 

Consequently, stories that were groomed on a detailed level may move to the "Scrap" section. That is okay. If the current product priorities indicate that this makes sense - so be it!

Don't bother tracking backlog that nobody will work on in the near future.

A good indicator of when you should weed:

You have items in the backlog that were not touched for more than 3 Sprints, and nobody has bothered asking about them.

Use frequent weeding to keep a significant, concise backlog.