Q&A: Our Customer says that our software is buggy?

Question: "When a stakeholder says to the Scrum team that our application has a lot bugs although the team has a good test coverage, what could be the team's take on that?" (source)

Dear Saad,

that's a good question.

The simple answer: Tests do not guarantee high quality software. 

Many Scrum teams feel that when their DoD is met and the Product Owner has accepted their Product Increment, they are "good to go". 

They might justify their behaviour based on whatever was agreed between them and the Product Owner. That might be quite revealing if the PO is the person who claims there are bugs. That would make a great team alignment&trust topic to discuss in a Retrospective. The key question here would be: "What does justification help - does it help us earn money?"

When another stakeholder (e.g, prospective customers) evaluate the team's success, this statement reveals that team has missed something important somewhere along the line. It might be in their approach, in their DoD - or in their understanding of stakeholder needs. 

We could now get into an argument of "defect" vs. "bug" vs. "feature", but that doesn't even make sense, because we'd still have the problem of a dissatisfied customer that we want to resolve.

How should you deal with that stakeholder?

Ask them to clarify what they mean with "a lot of bugs". Let them show you examples. Pay close attention. Learn how they use the product and what is important to them. Be open-minded.There's a good chance you made past assumptions that can't be held up any more. Try to discover what you can do different in the future. 

While we can never rule out the possibility of an insane stakeholder, they are rare. Most stakeholders want a good product, although their understanding of "good" may differ from yours.

Here are some items you may need to consider:

From technical perspective:

First, you could have written the tests "around" the defective areas - and second,  there's this difference between "works as designed" and "works as intended".

From customer perspective:

When you walk into a fastfood shop and get a Chicken Burger with a glove in it, you don't care how many hygiene checks ("tests") the company has - you care for what you experienced.

Keep in mind the Agile Principle #1 "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software".

This leads to two questions:

1. Why all of a sudden "a lot of bugs"? Wasn't that stakeholder involved frequently?

2. Has the team aligned their definition of value and quality with the stakeholder?

I leave you with are a few extra questions:

1. Do you have good tests? 
2. Is your test approach adequate?
3. What is your defect strategy?
4. What's your approach to communication debt?

There are answers, but you need to find them by yourself.

All the best,

Michael