The abuse of Cynefin

Scrum has been the tact giver for "Agile" for a long time. And Scrum is all about empiricism. And this "empiricism" has become a problem in recent years: Self-proclaimed, unqualified "coaches and trainers" proclaim that current organizational processes don't help, and hordes of incompetent "Agilists" swarm the market, only to wreak havoc on unsuspecting organizations.

Based on a mis-representation of Cynefin, people abuse Scrum and ditch available knowledge wholesale, because ... "In the Complex Domain, you don't know until you tried."

Now, is Scrum or the Cynefin framework really the cause of this problem? Not really - they are merely the door-opener for the snake oil sellers. And given Cynefin's and Scrum's easy appearance, people don't spot the trap until they fell for it. These frameworks are so popular and so easy to abuse, and it's really difficult for someone who sees them for the first time to discern what's correct and what is a mis-application.

Cynefin Framework by Dave Snowden

Now, let me describe the chain of "Cynefin Reasoning" that leads us down the wrong path:

"Knowledge Work (e.g. Development) isn't simple. It happens in the Complex domain. In the Complex domain, we have unorder where the relationship between cause and effect can only be perceived in retrospect and the results are unpredictable. Complex systems are dispositional and not causal. Hence, we cannot rely on good or best practices. Instead, we need to create safe to fail experiments and not attempt to create fail safe design."

It's a flawed understanding of Cynefin, combined with a false dichotomy that becomes a toxic soup. Here's why:

1. Just because something isn't simple doesn't mean it's automatically "complex". There's the entire domain of Complicated problems that's blissfully ignored, and even "Complex" is a category of varying degrees of complexity.
2. The idea that existing knowledge is entirely inapplicable doesn't describe the "Complex" domain - that would be the "Chaotic" domain: We can predict the outcome of a software development process pretty well. What we can't predict quite so accurately are customer reception and market conditions, but even there, we have quite elaborate mechanisms.
3. A shallow "dip into chaos" doesn't mean we should engulf, immerse or drown ourselves in chaos. Whenever we can prevent chaos, we're well advised to do so.
4. A scientific approach would rely on so-called "experimental conditions" where we fix all but the one variable under examination. If we let go of all control variables, we really won't be able to predict anything any more ("Chaos"). The latter is pretty pointless, and it can be avoided in knowledge work.
5. Just because something is "complex" doesn't mean we have no reliable process and are fully reliant on empiricism. To retain control, we need to minimize the impact of uncontrolled factors. For example, we would never have a smartphone if we didn't know exactly how to build computers, exactly how to build mobile networks, exactly how to mass-produce microtechnology, and those things would be a nightmare to use if we didn't know exactly how to build software that doesn't crash. It's complex, but it relies on a lot of things we precisely understand, and operating in this domain without high degrees of knowledge would be economic suicide.

And that's where we get into the mess today often called "Agile":

People don't understand what they're doing, because it's not a requirement to get into an "Agile" role. Fake "Trainers" without development experience make it look like that's not required - because, "hey, we got Empiricism". We encounter so-called "Agile Coaches" who don't know the building blocks of a functional company, we have to put up with 2-day "Scrum Masters" who don't know how to build a team, and see misplaced "Product Owners" who don't know what makes a product successful. And that's just scraping the surface.

Does this sound familiar?
"We don't need market research, probabilistic forecasting or demand control: Let's just write a User Story and put it into the backlog!" - "There's an entire science around Demand Management" - "Who needs that? Let's just incrementally Inspect and Adapt!"
Product Management, Quality Management, Process Management, Delivery Management, Operations Management, People Management, Finance Management, Sales Management ... we know a lot about these things! In the "Agile" community, however, there's a growing number of people who meet these with resentment, ideology and opinion: "managers are worthless anyways, so let's shoot all of that to the moon and apply empiricism!" 

The knowledge is discarded wholesale under the pretext of Cynefin - complexity and empiricism become the arguments against existing knowledge. "Inspect and Adapt" trumps science. Hence, "Agile" gets a free pass for obliterating functioning organizations: people with zero understanding of a domain "coach" long-term experts who have academically studied a subject, read the books, and gotten their share of field experience - and some even have the galls to claim that "coaching in the absence of knowledge works better, because it allows the coach to be unbiased!"

Thus, we set the playing field for quacks who will dismiss all scientific achievements and progress we have made in software engineering, bringing on the mumbojumbo, kumbayah, post-its and canned bikablo doodles: those professionals who did put in the hard work become indiscernable from quacks, organizations and individuals decline, order descends into chaos.

"Agile" has become a habitat for the same kind of science denialism as Anti-Vaxxers or Flat-Earthers: Except it's more subtle to spot, because the bodies of knowledge these people despise are the invisible fields in knowledge work. And Cynefin has been their door-opener.

Let me conclude with a question: "How will you approach complexity?"

Design Choices - is it really BDUF vs. Agile?

Do we really end up with Chaos or randomness if we don't design upfront?
The question alone should be sufficient to warrant that the answer isn't so straightforward. Indeed, it would be a false dichotomy - so let's explore a few additional options, to create choice.

Let's tackle the model from left to right:

Upfront Design

The epitomal nemesis of agilists, "Upfront Design" (UD) is the idea that both the product and its creation process can be fixed before the product is created. This approach is only useful when the product is well known and there is something called a "Best Practice" for the creation process.
In terms of the Cynefin framework, UD is good for simple products. A cup, a pen - anything where it's easy to determine a cost effective, high quality production approach. This is very useful when the creation process is executed billions of times with little to no variation for an indefinite time.

Normally, Upfront Design isn't a one time design effort. There are extensive prototyping runs to get the product right and afterwards, to optimize the production process before the real production process starts, as later adjustments are expensive and may yield enormous scrap and rework. All of this upfront effort is expected to pay off once the first couple million units have been sold.
The problems of this approach in software development are closely related to ongoing change: There is no "final state" for a software product except "decommissioned".
Your users might just discover better options for meeting their need in certain areas of the product - would you want to force them to do with a suboptimal solution, just because you've finished your design?

Likewise, a new framework could be released that might cut development effort in half, providing better usability and higher value at better quality: Wouldn't you want to use it? UD would imply that you'd scrap your past efforts in the light of new information and return to the drawing board.

Industrial Design

In many aspects, Industrial Design (ID) is similar to Upfront Design - both product and process design are defined before the creation actually starts. There is one major difference, though: the design of the product itself is decoupled from the design of the product creation process. This creates leeway in product creation based on the needs of the producer.

ID accounts for specifying the key attributes of the product upfront and leaving the creation process open until a later stage. The approach acknowledges that the point in time when the "What and Why" is determined may be too early to decide on "How". In implementation, ID accounts for "many feasible practices" from which appropriate "best practices" are chosen by the process designer for optimizing meta attributes such as production time, cost or quality.

Glancing at the Cynefin framework, ID is great for complicated, compound products assembled from multiple components in extensive process chains. Take, for example, a cell phone, a car or even something as simple as a chair, all commodities which get mass-produced, although both with a limited lifetime and still a limited quantity.

The challenges of ID in software development are similar to those of UD: there is no "final state" for software - and by the time it gets into users' hands, requirements might already have changed.

While for most industrially created products, the economic rule "Supply creates its own demand" applies and marketing and sales efforts can be used to make the available product attractive to an audience - Enterprise Software Design is about creating a product for one specific audience that doesn't like to be told what they're supposed to need!

Most corporate software development still happens in a similar way, although in many cases, the creation process has been defined and standardized irrespective of the product, with the assumption that all software is the same from a development perspective: This might be akin to assuming that a gummy bear follows the same creation process as a car or a chair!
Software development doesn't produce the same software a million times - we create it once and use it. And the next product is different. If indeed your company pays developers to develop the same product time and again, I have a bridge to sell you ...

Emergent Design

Emergent Design (ED) is buzzword terrain, as different people have a different understanding of what this means, and while advocates of UD/ID might call ED "No design", we likewise see people who indeed have no design call their lack of structured approach ED. This muddies the waters and makes it complicated to understand what ED is really about.

From a Cynefin framework perspective, ED is a way of dealing with the chaotic Domain in product development - when neither the best solution nor implementation method are known or constant! ED is what typically happens during prototyping, when various options are on the table and by creating something tangible, appropriate design patterns start to emerge.

ED requires high discipline. as systematic application of different design approaches helps reduce both time and effort in establishing useful designs, yet the designers need to fully understand that at any time, following a pattern may lead to a dead end and the direction needs to be changed. ED therefore requires high skill in design, quick forward thinking as well as massive flexibility in backtracking and changing direction.

ED accounts for constantly changing, evolving and emerging customer needs, a near infinite solution space and the inability of any person to know the next step before it has been taken.

Modern software development infrastructure is a showcase for ED: It takes minutes rather than days to change a design pattern, add, modify or remove behaviours and get the product into users' hands. The cost of change on a product level with good engineering practice is so low that it's hardly worth thinking upfront whether a choice is right or wrong - a design choice can be reverted cheaper than its theoretical validation would be.

The downside of ED is that neglect in both process and product design can slow down, damage and potentially outright destroy a product. It is essential to have a good understanding of design patterns and be firm in software craftsmanship practices to take advantage of Emergent Design.

Integrated Design

With Integrated Design (IntD), the focus shifts away from when we design to how we design. While UD and ID assume that there are two separate types of design, namely product and process - ED assumes that product design patterns need to emerge through the application of process design patterns.
IntD takes an entirely different route, focusing on far more than specification - it doesn't just take into account meeting requirements and having a defined creation process.

Peeking at the Cynefin framework, Integrated Design is the most complex approach to design, as it combines several domains into a single purpose: an overall excellent product.

IntD interweaves disciplines - given Cloud Infrastructure, development techniques such as Canary Builds are much easier to realize, which in turn allow the delivery of experimental features to validate a hypothesis with minimal development effort, which in in turn requires production monitoring to be an integral part of the design process as well. The boundaries between product design, development, infrastructure and operation blur and disappear.

Integrated Design requires a high level of control over the environment, be it a requirement modification, a software change, a process change, user behaviour change or a technological change.
To achieve this, practitioners of IntD must master product design techniques, software development practices, the underlying technology stack including administrative processes, monitoring and data science analytics. I call the people who have this skillset, "DevOps".

"Any sufficiently advanced technology is indistinguishable from magic" - Arthur C. Clarke

While master DevOps do all of these things with ease, people unfamiliar with any of the above techniques may not recognize that a rigorous approach is being applied and ask for a more structured, distinguishable approach with clear dividing lines between product design, process design and system operation.
Rather than accepting the near-magical results produced by such a team, they might go on a witch hunt and burn the heretics at stake, in the wake destroying the product built in this fashion.

And then, finally, we still have the default option, oftentimes used as the strawman in the advocation of each other design approach:

No Design

While seemingly the easiest thing to do, it's myopic and fatal to have No Design. No reputable engineer would ever resort to applying No Design, even if they know their design might be defective, expensive or outright wrong.

The reasons why No Design aren't an option go a long way, including - but not limited to:

• You don't know whether you achieved quality
• You're continuously wasting time and money
• You can't explain why you did what you did

The problem with No Design is that any design approach quickly deteriorates into No Design:

• An UD software product tends to look like it had No Design after a couple of years when people constantly add new features that weren't part of the initial design.
• When ID products change too much, their initial design becomes unrecognizable. The initial design principles look out of place and lacking purpose if they aren't reworked over time.
• Applying ED without rigour and just hoping that things work out is, as mentioned previously, indeed, No Design.
• While IntD is the furthest away from No Design, letting (sorry for the term) buffoons tamper with the an IntD system will quickly  render the intricate mesh of controls ineffective, to the point where an appearance of No Design indeed becomes No Design.

tl;dr

I have given you options for different types of design. The statement, "If we don't do it like this, it's No Design" is a false dichotomy.
This article is indeed biased in suggesting that "Integrated Design" is the best option for software development. This isn't a universal truth. An organization must have both the skillset and the discipline to apply ED/IntD techniques, otherwise they will not operate sustainably.

• No Design is the default that requires no thinking - but it creates a mess.
• Full-blown Big Design Upfront is a strawman. It rarely happens. Almost all software has versions and releases.
• Industrial Design is the common approach. It nails down the important parts and gives leeway in the minute details. 
• Emergent Software Design requires experience and practice to provide expected benefits. When applied haphazardly, it might destroy the product.
• Integrated Software Design is a very advanced approach that requires high levels of expertise and discipled.

Order vs. Chaos - a look at the science

Order creates a feeling of safety. Chaos, on the other hand, is a term with negative connotation.
But is order really what we want?

Using an analogy from material sciences, let's take a different look at the struggle of "order versus chaos".


Take a look at this model:

Order is the "frozen state" and Chaos is the "volatile state". I use this example on purpose: Kurt Lewin has proposed a 3-state change model, "Unfreeze-Change-Refreeze". It perfectly correlates to material sciences.

Lewin assumes that organizational structures and/or processes are best kept in a "frozen state". And in many organizations, that's true. Does it need to be true? Let's take a step by step look by examining the states first.

Different states

Frozen = Ordered

A frozen organization is ordered. It doesn't matter how effective or efficient the organization is - things are clear. Well, maybe not so much, but still. There are known structures to uphold, there are known people to address, there's a known protocol to follow and know processes to apply.
This makes things simple: If A, then B.
I go to work in the morning, know what I will be doing - and even when I take a week off, when I come back, I know the exact state where I will resume.

Terms used to describe a frozen state include: "predictable, reliable, convenient". All of these words are positively connotated both with workers and managers.

Hence, the desire to freeze the organizational system.

Along comes a troublemaker. A person who does things different. Who won't accept "That's how we always done it" for a reason. Who breaks rules to get stuff done. Who bypasses hierarchies getting in the way of success. Who messes with people following dysfunctional processes without further ado. Like - me.

In a frozen state, the organization will consider such a person a dangerous foreign body needing to be dealt with. A single person faced with frozen organizations will be forced into one of two choices: adapt - or get out. That's why change initiatives are well-nigh impossible without a strong guiding coalition as proposed by Kotter: It's not about doing things differently, it's about accumulating a critical mass of people with sufficient power to unfreeze the system before even bothering to really go on with the change agenda.

Liquid = Nonlinear

Getting out of a solid, frozen state in physics requires dissolving the bonds between system components. Depending on what you want to change, you may need to un-link structure, people and processes alike - simultaneously!

Dissolving links means that communication structures will no longer work as before, processes will no longer produce the same outcome - and results on any level may change. The bigger the incision, the more likely unpredicted side effects will occur.

People lose the feeling associated with an order state - predictablity wanes, reliability is reduced and people start experiencing the discomfort of needing to think and connecting dots differently. When essential, strong links haven't been dissolved, the system returns back to the former stable state as soon as possible and may even obtain resistance strategies - inoculation to ward off future change.

Going back to our physics analogy, we need to invest energy into an ordered system to unfreeze it. The amount of energy needed to make a permanent change in an ordered structure directly correlates with the strength of the links which need to be broken.

Let's take the example of water:
It takes a rather limited amount of energy to melt an ice cube.
With significantly more energy, we could split the molecular bonds and turn the ice cube into H2 and O2.
Should we desire to reform the very atomic bonds and turn the hydrogen atoms into helium, we need to invest a lot more energy into a much riskier and complicated process.

In either case, the former state will cease to exist - and that's where reactions such as fear, grief (Kübler-Ross) and entitlement come in.

Volatile = Chaotic

Volatile systems do not display the characteristics of an ordered system. The high energy inherent to each particle allows them to move rapidly - making it extremely difficult to predict the next state of even a single component in the system, much less enabling methodic control on the overall system.
In a chaotic state, change doesn't require an investment - it happens continuously and without trigger. Change isn't something that needs to be initiated, much rather, it would need to be channeled to produce something desirable.

In a volatile state, the system's drive to change is bigger than any component's capability to stabilize, hence no change is permanent.

A molecule in a gaseous state would not consider this state "anomalous" - much rather, it would need to be deprived of its energy before entering any other state.

And this is an important part of volatile systems - the components have high amounts of energy!

Like order is produced by draining energy from components, chaos is produced by energizing components.

State transitions

Material state transitions can be measured and predicted with high accuracy - something not quite as simple when dealing with an organizational system combined of a complex structure including many individual people, highly interrelated processes and potentially an innumerable amount of technological dependencies.

State transitions in a frozen organization are often considered undesirable, as people are afraid that something will get broken for good:

As the  "frozen" state of an organization becomes stronger, the more energy will be required to reach the "un-freeze" state which makes state transition possible. Combined with the complexity of the system, the amount of energy required for a successful defreeze might be roughly the same amount required to enter a volatile state - and depending on the size of the desired change, an excursion into a chaotic state may be required.

When being unfamiliar with the chaotic domain, there is both the danger that chaos gets out of hand or that when the system gets into a stable state again, that state is undesirable.

The false Order-Chaos dichotomy

Frozen systems are stable, but they aren't very malleable. The key characteristic of frozen systems is their lack of energy. An organization with stable structures that lack energy is in constant danger of being stuck in the wrong place - and won't be able to make the necessary move to remain sustainable.

On the other end of the spectrum, chaotic systems aren't stable - and equally un-malleable, albeit for a different reason. The key characteristic of volatile system is their high energy - and therefore, the impossibility to ever nail down anything.

It's not so easy to say whether order or chaos is better - that would depend on what you want to achieve. Based on the points above, we should realize that we're not forced to choose between order and chaos. The choice is a false dichotomy.
Just because water isn't ice, it doesn't mean that you'll die from steam burns.
It could equally be a refreshing glass of cool water or a warm cup ready to make some tea.

The idea "If it's not ordered, then it's chaos" is nothing more than a false dichotomy: There's a large spectrum of conditions between frozen and volatile - and while neither of the two extremes is specifically habitable, the range inbetween very well is.

Get used to change

When you look at a glass of water, you can't tell where every single molecule is. Even if you knew where it was five seconds ago, you can hardly tell where it will be in a few seconds. At the same time, you can tell a few things with fairly high confidence:

• You're looking at a glass of water
• The molecule in question is within that glass of water
• That molecule will still be within that glass of water in a few seconds.

You can even make more reliable predictions:

• If you throw a pebble into the glass, the water will (for the most) remain where it was.
• The pebble will not significantly affect the way your water behaves.
• When you take the pebble out, the water will look like it did before.

Interpreting the analogy, the liquid, non-linear state is both more flexible than the ordered and more reliable than the chaotic state. 

If you want to keep your organization robust to outside interference, you need to abolish the idea that everything needs to be in place: The ordered state can't deal with changing circumstances. 

In an ordered state, Lewin's "unfreeze-change-refreeze" process is essential to adapt.
In a non-linear state, there is nothing to unfreeze, and nothing in need of being re-frozen.

The practical interpretation

We talk a lot about "Agile Transformation", and in the minds of people this is unfreezing a non-agile organization, changing it towards an agile organization, then refreezing it in its agile state.
There is no such thing.

The agile organization is like water, constantly in a nonlinear state. Change in an agile organization isn't a project. Instead, everyone and everything within an agile organization is constantly subject to change.

Every person thinks about new, better ways to achieve things every day - every process can be scrutinized and modified at any time. When outward circumstances change, the agile organization doesn't start a massive adaption initiative, they just do what it takes to deal with the new circumstance.

And that's why you can't "buy agile" - to be agile, you must be in that liquid, transient state - at all times!

Agile learning for starters

I have previously discussed the "cost of learning" and it's impact on the learning strategy. After establishing that we should always keep this cost of learning below the Point of No Return, let us consider the differences in learning. The dogmatic statement "A coach should not prescribe a solution, but foster self-learning", presumes that self-learning is universally the best approach. But is it?

Let us consider which companies/teams typically call for help, based on this simple model:

Do you know why you don't know what you don't know?

There's a hidden relationship to the Cynefin Framework hidden: Software development is a socio-technological problem, and the issues of communication, understanding and skill are just a few factors affecting the team's performance. We work in the complex domain, where any model has an inherent error.
Usually, when a company requests external help, they tend to be basically aware that they don't really know what their problem is and that they assume someone else can help them make progress. In terms of our model, uncertainty is high and people admit that their specific knowledge and understanding of the problem domain is shallow. That's good. It's a basis for learning.

Initiaing problem solving

We have a wicked problem here: How do we know we're doing the right thing - and how do we know we're getting better at it?
A consultant has no choice other than first gaining clarity whether the problem is comparable to a problem where a solution is known, so would first try to drive down uncertainty - by asking questions and experimenting with the process.

If the problem is in a domain where deep expertise is available, the problem solving process is reduced to tapping into available expertise.

If the attempt to reduce the problem to a domain where a solution is known fails, this indicates that we're working in the domain of the Unknown.
This one splits down again:Either, we know that all known solutions fail, in which case we need to innovate - or all attempts to reduce uncertainty failed, which indicates our problem is ill-defined and we need to clarify the problem until we have a workable problem.

Innovative problem solving

If there is need to innovate, we're pretty much clear that we'll be using empirical data, feedback loops, inspect+adapt and experimentation to iteratively anneal the situation. The best thing a consultant can do in this situation is to provide support based on their own experience to discern which experiments make sense and what the available data implies.

There are tons of techniques for innovative problem solving, starting with Kaizen Events, Design Thinking, TRIZ ... potentially even a full blown Design For Six Sigma (not advised). Determining the suitable problem solving technique may also at the discretion of the consultant.

Introducing known solutions

When expertise is available, the consultant must factor in the impact and urgency of getting the problem solved.
Impact is high when there is a risk of crossing the Point of No Return, i.e. destroying the company / team, or have individuals lose life, health or their job.
Urgency is high when you only get one shot.

• If both impact and urgency are high, a dogmatic solution will save time at the expense that the inherent understanding remains low. Autonomous learning is purposefully replaced with prescriptive approach for a greater good.
• If impact is high, yet urgency is low, the consultant may choose to underline the solution process with moments of learning to deepen understanding. This will reduce long-term dependency and the risk for misunderstood assumptions around the solution.
• If the impact is low, yet there is a sense of urgency, the consultant might actually provoke "learning from failure" to create deep understanding for the next time.
• If both impact and urgency are low, the consultant should not invest further time. Providing a pointer on how the team could learn solving the problem can be sufficient. If they learn - good. Otherwise - no harm done.

Summary

In this article, I described only the consultant's approach when the team is lacking knowledge and ability that is available to the consultant. A different approach is required the team's knowledge already exceeds that of the consultant.

A good consultant weighs the costs of learning against the benefits of learning and chooses the optimal approach, carefully considering the tradeoff between short-term results and long-term results.

Innovative problem solving should generally not be used for known solutions, since that approach is inherently inefficient. Although it facilitates learning, it also maximizes the cost of learning.

Peoples who dogmatically insist on facilitating innovative problem to maximize learning solving might force the team to reinvent the wheel when a firetruck is sorely needed. That's not helpful. It's snake oil.
There are times for learning and times for just doing. Know the difference.

Are problems your problem?

Many organizations, and likewise the individuals working there, struggle with the notion of "Problems".
A "shoot-the-messenger" culture makes it impossible to name, much less claim ownership for, problems. To be able to still deal with reality, the very term "problem", inherently considered bad and a career-killer, gets sugar-coated or diluted. People start talking about "impediments", "issues" instead - and avoid doing that as long and much as possible.

Problems are not bad

Innately, a "problem" is just a "problem". It means that something does not work out as originally planned. This is not bad, it just means we have to change the plan - we might still reach the desired outcome.
The first step is to accept that "problem" is a neutral term, indicating "need for change" - neither good, nor bad.
 

Apples and ... Problems

Let's compare a problem to an apple. You can either get the value from the apple (eating it) or let it rot. Once you have the apple, you either take advantage of it, or it spoils.  That's totally up to you.
Treat your problems the same way: When you see one, try to get as much value out of it as possible. By ignoring it, your value will be Zero, but you will still be stuck with the cost of disposal.

Where do Problems come from? 

Can't we just make a better plan to avoid problems? Don't problems just reveal the incompetence of the planner? These are typical questions from a problem-averse company.
But we must accept that the world isn't simple. Problems arise not only because we aren't omniscient, but also because nobody's perfect and the world is changing. While some organizations expect their managers to be omniscient and everyone on their staff to be perfect, they still don't live in a bubble: The world around them is still full of imperfection and is changing at a rapid pace.
Because of this, we can't plan for everything - and even if we did, the plan would still fail. And that's a problem.

A problem is no problem

Many people consider it a flaw to admit having a problem. Nothing could be further from the truth: Being keen in observing discrepancies between plans and how they work out indicates a good analytic capability. The biggest concern is that in a complex system, a problem in one area may require adjustments in another area. Only by communicating that there is need for adaption and why this need arises will the entire system remain stable.

No problem is a problem

Denial of problems or being completely oblivious about problems in the surroundings is a typical response by those who have been conditioned by their environment. Sentences like, "In our organization, there are no problems" are common in companies where the assumption is that "sufficiently intelligent people can plan and anticipate everything". Not only are both denial and obliviousness problems - an environment where openness and integrity are not valued is another problem.
Usually, an environment where nobody sees a problem is the most unhealthy, because needed change is often procrastinated until too late.

Talk about problems

You can improve the effectiveness of yourself, your work - and your organization with a few simple steps:

1. Admit that problems exist. Call them what they are.
2. Don't shoot the Messenger. People who name your problems are not your enemy, they want to help you improve.
3. Live Continuous Improvement. Deal with problems before they grow out of proportion.

Failing to plan is planning to fail!

The Agile Manifesto boldly states, "We value ... Responding to change over Following a plan". I can't reiterate often enough that these are not mutually exclusive conditions.

I remember working in a project which coined the idiom "headless chicken mode" to describe the activities resulting from new feature requests: Everyone got busy, running into different directions - but nothing substantial resulted.

While panic and blind actions are response to change, they certainly are not the best. What the Agile Manifesto is concerned about is not plans in themselves - but that new information tends to make old plans obsolete!

Even seasoned developers occasionally fall into the habit of not planning, time and again! Throw out a (big) task that is fun and people will jump head in - without considering the long term objective! The result? When the window of opportunity closes, we have great ideas and fascinating code, but no marketable product!

Here are some things which should definitely be planned:

What do we hope to accomplish?

If you are uncertain as to what you will have accomplished when you are done, you will have no idea of how to know whether you have met the goal yet. It is completely legitimate to change your objective once new information uncovers that another course makes more sense.

It is not okay to be vague or unclear about your future accomplishment.

If you can not state your objective in one sentence - plan better!

In which order do we want to deliver what?

Agile approach suggests going MVP rather than perfectionism. Often, it takes a whole lot of thought to discover what is actually the minimum viable product and when we can deliver it.

The faster you can deliver something, the faster you will know whether you properly understood the need. Set short term goals that help you meet a long term goal. Don't go for that Big Hairy Audacious Goal without delivering value time and again in between!

If you have an ambitious plan that could take two months or more, don't set out unless you can break that plan into at least 3 steps that you can put into delivery order. Plan better!

How do we Inspect and Adapt?

When an idea is new, it tends to be coarse, ambiguous and with a lot of room for change. This uncertainty is no problem - unless we fail to manage it.
We can not remove uncertainty by drawing a straight line through uncharted territory - but what we can do: Take small steps, then carefully plan the next step.
Your Inspect+Adapt plan also needs to include how you will know whether you have made progress in the right direction - or not. Using direct customer interaction is as valid as some Big Data solution like Google Analytics or Omniture. Just make sure you choose the one that suits your needs.

If you can't describe your I+A strategy in one sentences, you have failed already! Plan better!

What do we need to succeed?

You should be clear what you will need to be successful. The Bible states, For which of you, desiring to build a tower, does not first sit down and count the cost, whether he has enough to complete it? Otherwise, when he has laid a foundation and is not able to finish, all who see it begin to mock him, saying, ‘This man began to build and was not able to finish'

You don't need to sign up to Christianity to understand the wisdom in these words. It is okay to miscalculate. It's also okay to miss something. But it's definitely not okay to not even bother figuring out whether you even stand a sporting chance!

If you can't name the top 3 success contributors and your strategy for obtaining them: don't start! Plan better!

Summary

This list of planning factors is by far not exhaustive, there are many more things to consider. In general, agile plans do not imply Gantt charts or big documents - but a clear, concise strategy for what you will do in order to succeed.

Your two most valuable plan items that need full clarity are:

• The "big plan", a.k.a. the Product Vision - where you want to be
• The next step

At any time you have uncertainty about those two things, stop what you're doing and re-plan!

Agile process models, Six Sigma and Cynefin

Back in my days as Six Sigma Master Black Belt, I learned and taught that every process can be modelled as a function. We write, on an abstract level, "y=f(x)". We use tools like SIPOC and Ishikawa Diagrams ("Fishbone") to model the relationship between inputs and outputs.

Since most processes have some variation, we accept that in practice, the function really implies "y=f(x+e)", where e is an error term within certain tolerance limits, typically impacting the accuracy of the model by no more than 5%. Because e is so small and we can't control it anyways, we ignore it in the model.

But what do you do when y=f(x) is simply an invalid assumption?

That's where agility comes into play.

Simple

The first difference between product manufacturing and software development is that product manufacturing is reproducible and repeatable. You know the exact process to apply to certain inputs to get the predetermined output. This is the "simple" world based on the Cynefin Framework.

But nobody in their right mind would build the same software the same way twice. Rather, if you have it, you'd use it. Or, you make some changes. Regardless. The output is not identical.

Complicated

What typically happens is that once a customer (user) sees a piece of software, they get ideas of what else they need. In this case, it's more like "y=f(x,y)": Given a result, the customer defines the new result. The process isn't reproducible anymore, because the output of each stage is input again. As a Six Sigma practitioner, I already had some issues with outputs that were process inputs: SIPOC becomes confusing, but it was workable.

At this point, it makes sense to work incrementally, because you can't know the final output until you have seen intermediate output. This is why agile frameworks typically apply backlogs rather than finalized requirement specification documents. We accept that the user may need to gain some understanding of the system to decide what has value and what doesn't. We also accept that the user may change their opinion on anything that has only been a vague idea yet.

Failure to implement checks and balances will move the team into the Complex domain within a couple weeks. Validation is not a onetime initial activity but must actually become stricter throughout the project lest errors accumulate and disorder leads straight into a Chaotic environment.
There is no way to reach the Simple domain, lest you acquire a good crystal ball that lets you accurately predict the future. (If you got that, you probably shouldn't develop software but play the stock market.)

Complex

You get into a mess when you only assume that the customer likes the result, but can't know for certain whether they do. The definition of quality may be unavailable until after the result is available to users. This scenario is fairly typical in web development, so as developers we have to guess what customers will accept. When users don't like something, we made an error. And there's no way of knowing that it was an error until we deployed the feature. We have to live with the fact that "y=f(x,y,e)" We can't eliminate the error term from our model, much rather, we have to accomodate towards the ever-present risk. We can only try to minimize risk by getting frequent feedback. The more time between two deployments and the more content within a single deployment, the more likely we made some critical error in the user experience.

Processes like A-B testing and continuous delivery become critical success factors. 

While you cannot completely eliminate the randomness, creating a high speed feedback mechanism, such as actively involving users in every minor detail produced, minimizes the effect of errors and effectively permits working similar to a Complicated environment.
The absence of control processes which deal with randomness may cause disorder to quickly shift the team into the Chaotic domain.

Chaotic

The worst thing that can happen is that yesterday's truth may be today's error. Customers may change their preferences on a whim. Imagine you work in a project where yesterday, you were requested to build a web portal for e-commerce, today the customer wants a document management system instead. Any plan is invalidated when you can't rely on your requirements, user stories or feature requests - or whatever you call them. Your model becomes a form of "y=f(e)", where neither x, you input, nor y, your previous output, are relevant indicators of success or failure.

This is where Waterfall projects with long planning phases may drift: By the time the plan is realized, demand shifted due to factors outside the project's sphere of control. An example would be a Waterfall team building a well-defined, perfectly fine php platform over the past 2 years meeting all business requirements, only to find out the newly hired CTO has just announced that all newly launched systems must be implemented in pure Java.

The only good news about the Chaotic domain is: You don't have to be afraid that it gets worse. Everything the team does may be waste.

The best way to deal with the Chaotic domain is moving away from it by delivering iteratively to minimize the effect of uncontrollable random on results. Frequent releases, no more than a couple years, provide the opportunity to move into the Complex domain.

Conclusion

The nicest statistical process model doesn't help when there is no feasible way to keep error terms under control or any model you come up with has relevant variables which can not be controlled. Six Sigma techniques for statistically modelling processes is useful when the process is repeatable and reproducible (good AR+R), but it won't get you far in a domain where R+R is neither given nor desired.