Agile vs Waterfall - the wrong question

22 years after the "Manifesto for Agile Software Development," the battle for Agile vs. Waterfall still isn't settled - despite many prominent publications, such as a recent HBR article, trying to "settle the debate."

Unfortunately, this debate can't be settled - because it starts with the wrong question. We will take a look at the question that needs to be asked.

Finding the right question

Most authors start with a question that doesn't bring us any closer to a sensible answer, quite often out of ignorance on what "Agile" is.

What "Waterfall" isn't.

Let's examine some definitions you may find floating around.

"Everything that's wrong"

Well, congratulations: with that definition, even adding too much salt to your breakfast eggs is Waterfall. Fun facts - even "Agile" doesn't prevent things from going wrong, and not even "Waterfall" projects always go wrong (would that mean that Waterfall isn't Waterfall because Waterfa ... argh, paradox!)

A sequential, linear approach from idea to completion.

That's crudely based on Winston Royce's paper from the early 70's - but face it: no company actually works like that, and even Royce wasn't suggesting that. Even "Waterfall," in industry common practice, is a continuous, incremental development approach. Just with very big increments, and extremely slow feedback cycles (it's not unrealistic that it could take 2-5 years between when a requirement is defined until people get the thing they actually need.)

A stage-gated approach to product development.

That definition has two fundamental flaws:
First, you can stage-gate "Agile" approaches as well, and before 2006 (i.e. five years after "Agile") - stage gating wasn't even a thing in most enterprises: You can well do non-Agile projects without stage gates.

You can find another, much more comprehensive article on other things that "Waterfall" is not referenced here.

So we're in a bit of a pickle: We can't even find a non-strawman definition of "Waterfall" that would go beyond the completely fictional definition used in Royce's paper for how to not do things.

What "Agile" isn't.

Let's do the same favor to "Agile" and pull out some of the common misleading beliefs floating around:

Scrum is not Agile

Hard to overemphasize this, but there are two core problems with equating Scrum and "Agile."
First, there's a boatload of terrible Scrum out there that not only doesn't qualify as "Agile," it hardly qualifies as professional.

Second, there are many "Agile" approaches out there, and even though Scrum dialects are the most common, quite a lot of agile companies don't use even Scrum.

Inadequate processes, plans, documents, or contracts

This definition of "Agile" only rings true for illiterate people who can't make sense of the statement "While there is value in these things ..." - Agile approaches make use of these things. Any argument against "Agile" referencing to Agile's inability to structure themselves is either a strawman or ill-informed.

Lack of direction or tracking

Humorously, if that's the definition of "Agile," it's not a stretch to say that most large Enterprises are indeed extremely Agile.

On a more serious note: most professional Agile practitioners will emphasize the importance of clear goals, as well as frequently checking progress on these goals.

We're stuck with the same issue: if we use a strawman definition of "Agile," then everything sensible is a better choice - but none of these definitions hold true to the intent of the Manifesto.
The best definition I can come up with today is based on the very first line of the Manifesto itself: "We are discovering better ways of developing software by doing it and helping others do it." My definition, hence, is "The ability to do the right thing right, at the right time." And with that definition, everyone who proposes anything other than "Agile" - would have to argue why it would be preferable to do the wrong thing, do things wrong, or have poor timing: I find that case extremely hard to make.

Note that this definition of "Agile" says nothing about Frameworks, Methods, Roles or any other form of rules. It doesn't even say anything about values, principles or goals. It's so broad and encompassing that the dividing line is only, "Is that right?"

So - if nobody in their right mind would purposefully do something wrong, why is "Agile" even a thing?
And that leads us to ...

The actual choice

After this introduction, we realize that choice isn't Agile versus Waterfall. It's about answering the question how we deal with things going awry. But: why do things go wrong? Let's take Mark Twain's opinion here:

This provides us a powerful reframing of the entire debate: The "things we know for sure that just ain't so" are - unvalidated assumptions. With this framing, we can rephrase the question to:

How are we dealing with assumptions?

The fundamental choice is: do we proceed based on predetermined assumptions, or do we capture evidence to validate our assumptions? And at which point do we do this?

The framing thus stops being the unfortunate false dichotomy of "Agile vs. Waterfall," it turns into:

Which mechanisms do we employ when reality contradicts our assumptions?

A classical plan-driven approach is built on three core assumptions: we are doing the right thing, we know how to do it right, and we can determine the right timing.

A classical "Agile" approach is built on the core assumption that there is uncertainty: we don't know what the right thing is, we could be doing something wrong, and the world doesn't always follow our timing. The concept is the "VUCA World."

These approaches are both extremes, and in practice, there's always a form of compromise:

• We can start with a hypothesis of what's the right thing, and when it turns out that it isn't, we need to change what we do.
• We can define how we do things, and we have to change that plan when it no longer works.
• We can schedule when we do what, and when we miss that schedule, we need to re-schedule.

Every project manager who learned their ropes will agree that even in a classical, plan-driven approach, this happens all the time. Likewise, every Agilist will concede that we need a product vision, some working agreements, and a bit of forecasting. We can also agree very easily that we need frequent inspection points and to check whether we're on track, and to change direction when that's not true. So - where does the disagreement come from?

Ultimately, we're talking about the probability that an assumption we made is wrong. How likely did we to bet on the wrong horse? How likely are we riding a dead horse? And how likely is the race not going as planned?

Statisticians understand that probabilities aren't 0 or 1, but somewhere in-between. Depending on how likely an event is, we should use practices appropriate to deal with the uncertainty. And depending on how far away an event is, the less likely we can predict it. Hence, the actual question becomes:

How do we deal with uncertainty?

The answer to this question is extremely contextual. We only know that rigid practices without flexibility will be more problematic when we ran into something that we didn't expect - and being so flexible that we can't make any form of prediction isn't sensible, either.

Summary

The question that we should be asking is how much it will cost us to learn that we are wrong? When the answer is tolerable, we are doing okay. Otherwise - we are gambling.

And in some cases, it could be the most wrong and costly thing to not have a detailed long-term plan. Even though that's not considered to be very "Agile."

16 misconceptions about Waterfall

Ok, Agilists. It's 2021, and people are still using Waterfall in corporate environments. With this article, I would like to dismantle the baloney strawman "Waterfall" that's always proclaimed as the archenemy of all that is good and would encourage you to think about how exactly your suggested "Agile" is going to do better than the examples I have taken from real-world, professional Waterfall projects.

Here are some things that many agilists may have never experienced in Waterfall projects. I did.

What you think Waterfall is, but isn't

There are numerous standard claims about what's wrong with Waterfall, which I would generously call "statement made from ignorance," although there could be more nefarious reasons why people make these claims. Point is: many of the common claims are not generally true.

Big Bang vs. Incremental

Waterfall doesn't mean that until the determined end date of the project, there will be nothing to show. I remember when I stated that I worked in a 5-year Waterfall project, people from the Agile community called that insane. It's not. We had a release every 3 months. That means that the project had a total of 20(!) Increments, each with its own scope and objectives: Yes - Waterfall can be used to build products incrementally! In corporations, that's actually normal.

Upfront Design vs. Iterative Design

With each delivery, project managers, analysts and business people sit together and discuss the roadmap: which requirements to add or remove, and which priorities to shift. I have once worked in a product that was created in pure Waterfall for almost 20 years, and nobody could have anticipated the use cases delivered in 2010 when the product's first version hit the market back in 1992. Even Waterfall projects can iterate. Especially for enterprise systems.

Death March vs. Adaptivity

When you think that someone sits in a closet and produces the Master Plan, which must be slavishly adhered to by the delivery teams, you're not thinking of a properly managed Waterfall project. While yes, of course, there is a general plan, but a Waterfall plan gets adapted on the fly as new information arises. Timelines, staffing, scope, requirements, objectives - are all subject to change, potentially even on a weekly basis if your project manager is worth their salt.

Fixed Scope vs. Backlog

If you've ever done Project Management, you know pretty well that scope is very malleable in a project. When an organization determines that meeting a fixed timeline is paramount, Waterfall fixed time projects can be pretty similar to Sprints in managing scope. While of course, you get problems if you don't manage the Critical Path properly, that's not a Waterfall problem - it's carelessness. 

Fixed Time vs. Quality

Probably one of the main complaints about Waterfall is that a team delivering on a fixed schedule will push garbage downstream to meet the timeline. Again, that's not a Waterfall issue - it's a "fixed time" issue. If you flex the time, and fix the work package, there's nothing inherent to Waterfall that implies a willful sacrifice of quality.

(And, as a witty side note - if you believe that fixed time is the root cause for low quality: how exactly would Scrum's Sprint timebox solve that problem?)

Assumptions vs. Feedback Learning

Complex systems serving a multitude of stakeholders are incredibly hard to optimize, especially when these stakeholders have conflicting interests. The complexity in Waterfall requirement analysis is usually less in trying to get a requirement right, as it is in identifying and resolving conflicting or wrong demands. The time spent upfront to clarify the non-developmental interferences pays off in "doing the right thing." Good analysts won't be making wild assumptions about things that could potentially happen years down the line. When a release is launched, good Waterfall projects use real user feedback to validate and update the current assumptions. 

Handovers vs. Collaboration

Yes. There's something like stage-gates in most Waterfall projects. I myself have helped Waterfall organizations implement Quality Gates long before Scrum was a thing. But it's not inherent to Waterfall - otherwise it wouldn't have been a thing in the early 2000's. Also: don't misunderstand gates. They don't mean that an Unknown Stranger hands you a Work Package which you will hand over to another Unknown Stranger at the next Gate. What typically happens: As soon as analysts have a workable design document, they'll share it with developers and testers, who take a look, make comments and then meet together to discuss intent and changes. Good Waterfall organizations have collaboration between the different specialists whenever they need to.

Documentation vs. Value Creation

A huge misconception is that "Waterfall relies on heavy documentation" - it doesn't, depending on how you operate. Heavy documents are oftentimes the result of misfired governance rather than caused by the Waterfall approach itself. It's entirely feasible to operate Waterfall with lightweight documentation that clarifies purpose and intent rather than implementation details, if that's what your organization is comfortable with. Problems start when development is done by people who are separated from those who use, need, specify or test the product - especially when there's money and reputation at stake. 

Process vs. Relationships

As organizations grow large, you may no longer have the right people to talk with, so you rely on proxies who do a kind of Telephone Game. This has nothing to do with Waterfall. A good Waterfall Business Analyst would always try to reach out to actual users, preferably power users, who really know what's going on and build personal relationships. As mutual understanding grows, process and formality becomes less and less important, both towards requesters and within the development organization - even in a Waterfall environment.

Resource Efficiency vs. Stable Teams

There's a wild claim that allegedly, Waterfall doesn't operate with stable teams. Many Waterfall organizations have teams that are stable for many years, in some cases, even decades. Some of the better ones will even "bring work to the team" rather than assigning work to individuals or re-allocating people when something else is urgent. The "Resource efficiency mindset" is a separate issue, unrelated to Waterfall.

Big Batch vs. Flow

Kanban and Waterfall can quite well coexist. Indeed, I have used Kanban in a Waterfall setting long before I first heard of Scrum where requirements flowed through three specialist functions, and we had an average cycle time of less than one week from demand intake to delivery. Waterfall with Small Batches is possible, and can perform exceptionally well.

Top-Down vs. Self-Organized

I've worked with corporations and medium-sized companies using Waterfall, and have met a lot of Project Managers and Team Leads who have worked in a fashion very similar to a Product Owner: taking a request, discussing it with the team, letting the team figure out what to do how and when, only then feeding back the outcome of this discussion into the Project Plan. Waterfall can have properly self-organized teams.

Push vs. Pull

Whereas in theory, Waterfall is a pure "Push"-based process, the field reality is different. If you have a decent Waterfall team lead, it will basically go like this: We see what work is coming in, we take what we can, and we escalate the rest as "not realistic in time", and get it (de-)prioritized or the timeline adjusted. De facto, many Waterfalls teams are working pull-based.

Overburden vs. Sustainable Pace

Yes, we've had busy weekends and All-Nighters in Waterfall, but they were never a surprise. We could anticipate them weeks in advance. And after these always came a relaxation phase. Many people working in a well built, long-term Waterfall project call the approach quite sustainable. They feel significantly more comfortable than they would be under the pressure to produce measurable outcomes on a fortnightly basis! Well-managed Waterfall is significantly more sustainable for a developer than ill-managed Scrum, so: Caveat emptor!

Resources vs. Respect

Treating developers as interchangeable and disposable "resources" is an endemic disease in many large organisations, but it has nothing to do with Waterfall. It's a management mindset, very often combined with the cost accounting paradigm. The "human workplace" doesn't coincide well with such a mindset. And still, the more human Waterfall organizations treat people as people. It entirely depends on leadership.

Last Minute Boom vs. Transparency

Imagine, for a second, that you would do proper Behaviour Driven Development and Test Driven Development in a Waterfall setting. I did this in one major program, delivering Working Software that would have been ready for deployment every single week. If you do this, and properly respond to feedback, Waterfall doesn't need to produce any nasty surprise effects. The Last Minute Boom happens when your development methodology is inapproprate and your work packages are too big, not because of Waterfall.

All said - what then is, "Waterfall?"

"Waterfall" is nothing more and nothing less than an organized, sequential product development workflow where each activity depends on the output of the previous activity.

There are really good uses for Waterfall development, and cases where it brilliantly succeeds. It's incorrect to paint a black-white image where "Waterfall is bad and Agile is good", especially not when equivocating "Agile" to a certain framework.

Proper Waterfall

A proper Waterfall would operate under the following conditions:

1. A clear, compelling and relateable purpose.
2. A human workplace.
3. A united team of teams.
4. People who know their ropes.
5. A "facts are friendly" attitude.
6. Focus on Outcomes.
7. Continuous learning and adaptation.
8. Reasonable boundaries for work packages.
9. Managing the system instead of the people.

All these given, a Waterfall project can have a pretty decent chance to generate useful, valuable results.

And when all the above points are given, I would like to see how or why your certain flavor of "Agile" is doing better.

My claim

I challenge you to disprove my claim: "Fixing the deeper mindset and organizational issues while keeping the Waterfall is significantly more likely to yield a positive outcome than adopting an Agile Framework which inherits the underlying issues."