SysOps and Development: An Antipattern

Developers develop, Sysops operate. Obvious.
Well, that's classical thinking.

The consequence is that problems do not get resolved on time: Users suffer, value is lost and the reputation of the IT department is tarnished.

Obvious problems often don't get resolved in a timely manner - the loser is: everyone!

What you see above is real data taken from an organization where some Scrum teams are responsible for creating features and another team is responsible for operating the platform.

Just taking this example, the biggest current issue is an encoding problem - something that happens in the "real world", but not in a controlled development environment: Continuous Integration was successful, a feature hit the market - and boom!

Separate Responsibilities

In the example above, developers are busy churning out new features while SysOps are busy with fire-fighting.
Taken from the chart, it's been 3 days since the problem started - 3 days where SysOps are constantly stressed while developers are busy with stacking more features on top of existing problems.

XP proclaims "A Red Master is developers' Priority 1" --- well, whose priority 1 should a Red Production be? SysOps or everyone?
In a classic Kanban system, when the production line goes down, it should stop the entire machinery until the issue is resolved. This also holds true for agile teams: If anywhere in the organization, there is a problem, stop. Fix.
Don't go on producing more problems.

Perverse incentives

Developers are appreciated for delivering new features, while SysOps are appreciated for keeping the platform stable. Consequently, Devs prefer to work on new stuff rather than fixing existing stuff. Unfortunately, the organization as a whole suffers. Not rowing in the same direction can't yield any better results.

The very fact that people have different incentives here implies is a problem with autonomy and self-organization!
You must align the incentives of everyone working on the product, not only "development team".

Definition of "Done"

Regarding developers higher for delivering a new feature than making an old feature stable yields the problem above: "Just add a new story into the Product Backlog and we'll deliver it with the next sprint." - while this is a classic approach to Agile Software development "There are no bugs, missing Acceptance Criteria are new stories" and Scrum's "The Sprint is Closed towards modification".
This is a severe misunderstanding: Can you actually consider a story "Done" when it causes problems to the customer?

When you see that there's an operative problem with a new feature, it means that in your retrospective, you should put your DoD under scrutiny.

DevOps: A solution

As the agile proverb goes,  "You build it, you run it."
As long as developers do not feel the operational pain, they are not interested in removing it. Likewise, as long as SysOps are unable to fix the root cause of a problem in source code, they become apathic towards code-related problems.
Moving both the operative problem as well as the solution space into the team's sphere of responsibility, developers will take an interest not only in the code they write and how it meets the acceptance criteria, but also how the code actually performs in the real world.

Organizationally, this can be done as simple as moving a SysOp into the development team and then holding the team accountable for events on the productive platform.
The Developers become Dev-Ops who will come up with innovative solutions to detect, analyse and prevent problems. A DevOps can solve issues in a fashion that would be completely unthinkable for a SysOp who is confronted with a "build ready" black box software.




Side note
The above screenshot was taken from Medullar, towards which I, personally, am completely biased. I am one of the core developers of this easy to use, open source, cloud-based, universal monitoring and problem solving solution which was designed to require minimum levels of intrusion and resource capacity while offering maximum flexibility.
Medullar allows you not only to detect and analyse problems in any kind of environment. It provides an API and UI to remedy them in real time. It even comes bundled with it's own test framework, so you don't need anything else to improve your operative performance.

What's better: Scrum, Kanban or XP?

The question "Which agile framework is the best?" is often raised by people who are fairly new to the world of agile frameworks. In short: I can't answer this question without asking a counter-question: "Best for what?"

 

 

When your only tool is a hammer, every problem becomes a nail!

Let me analyze the problem in regards to the image above.

Scrum: keep things in place

Scrum is primarily known for the sprint cycle process and the ceremonies: Grooming, Refinement, Planning, Daily Standup, Review and Retrospective.
Scrum discerns only the roles Product Owner, Scrum Master and Developer. You need a Product and Sprint Backlog consisting of Stories, in the latter broken down into Tasks.
You'll need to define suitable Working Agreements, such as a Definition of Ready and Definition of Done with your team, then use Inspect+Adapt on these rules to optimize your process continuously.

Scrum is quite rigorous. If you do not have any of these things, you end up with a disdained "Scrum-But". If you do something beyond, you usually also end up breaking your Scrum.

Scrum is a wonderful framework for creating structure, clarity and order. Additionally, Scrum reduces volatility, enhancing predictability and confidence.

Kanban: Create flow

Kanban is often confused with the Kanban board, the most notable artifact. Making work visible, creating a smooth flow of work as well as identifying and removing bottlenecks are all key factors of successful Kanban.

Kanban is very flexible. You can start with Kanban wherever your organization is, without interference to structure or process. From there, you start optimizing.

Kanban is a wonderful framework for making impediments visible and optimizing delivery.

XP: Professional Craftsmanship

Extreme Programming is mostly concerned with Engineering Practices. Techniques such as Pairing, Test Driven Development, Continuous Integration and Refactoring will lift your developers to new levels of excellence.
Design concepts such as Whole Team approach, Collective Code Ownership, Clean Code, Emergent Design will broaden your team's horizon and capacity.
Lean concepts such as sustainable pace, Metaphors, Small batches, Standards , Automation and Continuous Improvement reduce waste and risk.

XP is process-agnostic. You can pick individual practices from XP that solve specific needs in your team, although taking the batch is advised for best results.

Summary

None of these agile frameworks is "better". Each of them has a specific area of use. As such, the frameworks become "better for solving a specific problem". Therefore, you need to define your problem first before answering the question "Which framework is best?" - a hammer is better for fixing a nail, but not for a screw. Of course, you can cut a board with a screwdriver, but a saw would suit your need better.

Take some time, define your specific problem, then work from there.
Here are some rules of thumb:

• Solve organizational or process issues with Scrum.
• When you can't seem to get anything "done", try Kanban.
• When your perceive quality or architectural issues, apply XP.

These frameworks are not exclusive. You can mix them - and high performance teams combine them. Scrumban is famous, for instance.

Disclaimer: None of these frameworks will actually make you successful in the long term without an Agile Mindset. If you have issues with Agile Values or Agile Principles, your only fix is returning to the Agile Manifesto - regardless of which framework you're using!

Why Test Driven Development?

TDD is sometimes proclaimed as a means for early discovery of all defects. Sometimes, people who are new to Software Craftsmanship practices wonder what tools they should use for TDD.

What TDD is not about

TDD is not so much about what tool you are using. TDD is tool-agnostic. The best way to do TDD is to use the same programming language. If you're looking for tools, you look in the wrong direction.
It's not about having a high test coverage. It's not about minimizing quality risks. In fact, it's not a Quality Assurance technique at all. If you use TDD for finding bugs, you're definitely on the wrong track.

What TDD is about

The key to successful TDD is to produce code in a fashion that each component is fully inspectable down to the most granular level, so that:

* You have a good software design

* You fully understand what each component, method and API does

* You minimize overhead of future reverse-engineering

* You lay a basis for easy changes, so that now you don't need to waste time on thinking about possible future changes

* If you need to change something fundamentally, you know where to best do it

* If something doesn't work as intended, you can isolate the root cause quickly

Summary

The main reason to employ TDD is not if you have "lots of defects". Chances are that you will not only not reduce the defect count with TDD on a poor quality legacy application - but that you will invest a lot of effort into increasing coverage without getting any measurable benefit to the user (customer) at all.

The best reasons to employ TDD are: when developers complain that they can not make changes to the software easily or they complain that finding the root cause of defects takes forever.

TDD is architecture+design topic.
The first minute you start doing TDD, each new line of code written, you will be working towards a better software design. It definitely pays off in the long term. 
Just don't look at the release defect metric, because you will not see the benefit there.

The five key ingredients for successful software development

All you need is good code that meets the requirement and you're set? That is short-sighted!

You need to keep a keen eye not only on the following key processes and artefacts to be successful in the long term.

Working Software (Code)

That's obvious. If you don't have a product, you have nothing.
This not only includes what the customer sees, but also what is going on behind the scenes. The statement, "A kitchen where someone cooks is always messy" might apply to the developer's office, but it must not apply to the code.

Smelly code costs real money and binds capacity that would be better focused towards innovation and progress.

Test Suite

To understand whether your product works correctly and does the right thing, you need a proper test suite. I radically proclaim "If your test isn't at least as clean as your productive code, don't bother".
A common (anti-)pattern I observe is that organizations de-prioritize testing in order to meet certain goals. Or, they actually put an emphasis on testing, but approach testing wrongly. The consequence? Flaky or unreliable tests that cause more trouble than they solve.

The money invested into testing is misspent if your tests are not well-managed, clean and comprehensible. However, abolishing your test suite equals abolishing your investment into sustainability. Don't ever reach this point.

From day 1 of a new product, invest into a proper test suite. Treat your test suite as the Holy Grail of long-term sustainability.

If you have existing software with that, go seek that Holy Grail - today!

Working Build

To actually reach Continuous Integration - or beyond, you need a smooth, swift build process. You should be able to modify your build as easily as you can modify your product's features.

I see many teams and companies who under-emphasize their build process. A sad fact is that they all run into build problems sooner or later. Some can't manage their dependencies any more, some don't understand why their build is unstable - and the build becomes a nightmare for every developer!

Like you manage your code and the test suite, you must manage your build.This includes the build management system, the build artifacts as well as the build infrastructure.

Environment

A good product is not impeded by the environment, which includes technical infrastructure as well as corresponding tools and integrated components.

An overly complex or under-performing environment are equally detrimental to the success of your product.

For example, the wild growth of turnkey service solutions is the best way to lose control of what is actually going on. It is quite easy, for instance, to include a tracking or ad service. It is, however, incredibly hard to debug your software if such an external service is causing a malfunction in your product.

Another example, if you are using a myriad of virtual machines without having a proper hardware management will cause tremendous trouble when trying to find out what went wrong where, and why. The time lost to a poorly managed environment can quickly exceed the time you are actually spending on the product you're building.

 

Runtime

"Works on my machine" - isn't it a running gag?

How do you control that the product also works in production? Of course, you need monitoring and feedback.

But technical monitoring doesn't cut it.  Have you ever taken a detailed look into your log files? Whenever I engage with a new client, I take the liberty of scanning the productive logs for stack traces and the other obvious stuff ("Exception", "ERROR", "CRITICAL", "FATAL"). Doing that only takes a minute with grep and I usually find a myriad of stuff for the next retro.

Sometimes, the software doesn't display any obvious sign of malfunction for your (Dev-)Ops while thousands of users around the world have no clue why they can't reach their intended goal.

To understand and control your runtime properly is a tremendously important - yet oftentimes underestimated - success factor.

Conclusion

You must take care of five critical ingredients:

Code, Test, Build, Environment and Runtime.

Give sufficient attention to all of these will drastically reduce your risk of failure.
Ignore one of them and you will find yourself struggling to keep customer satisfaction and progress up.

Failing to plan is planning to fail!

The Agile Manifesto boldly states, "We value ... Responding to change over Following a plan". I can't reiterate often enough that these are not mutually exclusive conditions.

I remember working in a project which coined the idiom "headless chicken mode" to describe the activities resulting from new feature requests: Everyone got busy, running into different directions - but nothing substantial resulted.

While panic and blind actions are response to change, they certainly are not the best. What the Agile Manifesto is concerned about is not plans in themselves - but that new information tends to make old plans obsolete!

Even seasoned developers occasionally fall into the habit of not planning, time and again! Throw out a (big) task that is fun and people will jump head in - without considering the long term objective! The result? When the window of opportunity closes, we have great ideas and fascinating code, but no marketable product!

Here are some things which should definitely be planned:

What do we hope to accomplish?

If you are uncertain as to what you will have accomplished when you are done, you will have no idea of how to know whether you have met the goal yet. It is completely legitimate to change your objective once new information uncovers that another course makes more sense.

It is not okay to be vague or unclear about your future accomplishment.

If you can not state your objective in one sentence - plan better!

In which order do we want to deliver what?

Agile approach suggests going MVP rather than perfectionism. Often, it takes a whole lot of thought to discover what is actually the minimum viable product and when we can deliver it.

The faster you can deliver something, the faster you will know whether you properly understood the need. Set short term goals that help you meet a long term goal. Don't go for that Big Hairy Audacious Goal without delivering value time and again in between!

If you have an ambitious plan that could take two months or more, don't set out unless you can break that plan into at least 3 steps that you can put into delivery order. Plan better!

How do we Inspect and Adapt?

When an idea is new, it tends to be coarse, ambiguous and with a lot of room for change. This uncertainty is no problem - unless we fail to manage it.
We can not remove uncertainty by drawing a straight line through uncharted territory - but what we can do: Take small steps, then carefully plan the next step.
Your Inspect+Adapt plan also needs to include how you will know whether you have made progress in the right direction - or not. Using direct customer interaction is as valid as some Big Data solution like Google Analytics or Omniture. Just make sure you choose the one that suits your needs.

If you can't describe your I+A strategy in one sentences, you have failed already! Plan better!

What do we need to succeed?

You should be clear what you will need to be successful. The Bible states, For which of you, desiring to build a tower, does not first sit down and count the cost, whether he has enough to complete it? Otherwise, when he has laid a foundation and is not able to finish, all who see it begin to mock him, saying, ‘This man began to build and was not able to finish'

You don't need to sign up to Christianity to understand the wisdom in these words. It is okay to miscalculate. It's also okay to miss something. But it's definitely not okay to not even bother figuring out whether you even stand a sporting chance!

If you can't name the top 3 success contributors and your strategy for obtaining them: don't start! Plan better!

Summary

This list of planning factors is by far not exhaustive, there are many more things to consider. In general, agile plans do not imply Gantt charts or big documents - but a clear, concise strategy for what you will do in order to succeed.

Your two most valuable plan items that need full clarity are:

• The "big plan", a.k.a. the Product Vision - where you want to be
• The next step

At any time you have uncertainty about those two things, stop what you're doing and re-plan!

Being a valuable Product Owner

In the Product Owner training, we learn that the Product Owner is in charge of the Backlog, and therefore controls "What" is being done "When". With this, the Product Owner leads Product Value - and: success.

Those are the basics.

Many times, you see "Fake Product Owners" in corporations. Retrained team leaders, project managers or even analysts.

They are handed requirements or, even worse, detailed specifications by other departments and are then expected to utilize the team to turn these into Working Software.

Let me be blunt here: If you have a good team, you don't need them. Because they can't do the job they've been set to.

It is not a Product Owner's job to play "Chinese Whispers" between business and the development team.

Understand the Customer

If you want to be a good Product owner, you must understand your customer. Please ask yourself a few questions about the product:

• Do you know any real customer and what they think?
• Do you really know what your customers are and will be doing with the product?
• Do you have facts, rather than opinions, regarding what the customer pays money for?
• Can you correlate features with relevant business figures? (more on this later)

A witty side reminder to PO's working in corporations: Unless you are working on an in-house product which is exclusively being utilized by staff of your company, your customers are not internal stakeholders, such as your Division Head, your Marketing Department, your Legal Department or your CxO.

If you do not have any access to real users and their needs, you are, at best, a proxy. Most likely, you'll be doing a political project where success is not determined by hard cash - which means, you won't be able to contribute anything relevant.

 

Determine Priority

Your team may be doing a tremendous job, but they are not omnipotent. They can't do everything at the same time, and they probably also shouldn't.

You probably heard the proverb, "Get your priorities straight".

Let me tell you: It's wrong. You don't have priorities. You have exactly one priority. Your first priority is "to satisfy the customer through early and continuous delivery of valuable software".

To do this successfully, you must be able to determine what satisfies the customer most, so that you can set your team to work on the one thing which provides the most value to the product.

Weeding the Backlog

You have a backlog rather than a requirement list mainly for one reason: You already know that the amount of requested work is higher than the amount of available resources.

Don't even bother with anything that can not be delivered in the near future, also don't even bother with anything that won't satisfy your customers.

 

You must say "No" to anything which you do not consider sufficiently valuable. Don't beat around the bush. Let people know "No means no." Please refer to "Weeding the Backlog" to see one approach to obtain a thinner backlog.

If you can not say "No" to anything put on your desk, you will be doing a miserable job. 

Learning how to say "No" so that people will not be personally offended with you is an art, however.

Slicing to the Bone

I love hate it when people approach me with "world hunger type" feature requests. Yeah, I understand that you want a widget that has thingummywat to do oomlah. But Rome wasn't built in a day.

Stakeholders love to push Epic work into your team, because they feel that they get something significant. Let me tell you what: Don't bother unless they can cut it down.

It isn't sufficient to merely throw a feature over the fence, groom it, develop it. That's not your job. Your job is to extract the Minimum Viable Product from each and every feature!

If you can't fit it into a single sprint, you are probably not working on an MVP.

To give you a specific example of what I mean: I can produce a skeleton CRM System with a single team in one sprint. But it won't do much. It will just tell me (and my customers) what must be done next to bring the most value.

Cut out everything that's not vital. Maybe that means cutting out the entire backend - maybe it means cutting out the entire frontend. But you must get useful, working software fast.

Keep the time!

Based on your customer, the priority and with minimal slices, be prepared to deliver what the customer wants, when the customer wants - with high quality. The only thing you must never promise is that the customer gets the entire scope they might have imagines.

Never, ever, postpone deadlines to stuff in more features.

I recently saw from a DevBlog for a new video game: "We could have made the skill wheel in 3D, but it would have cost too much development time." - throw out the chaff, never get yourself into a mess where you have to stand empty-handed before your customer because your scope didn't fit the time.

Your customers will be forgiving on the scope, but not on not having anything on the agreed date. Deliver. On time. 

Increment afterwards as long as there is value to harvest.

Summary

You are responsible to deliver a satisfying, valuable product in short, frequent intervals. If you do this well, the customers will be praising your product and your team - and you.

If you don't do this well, you'll quickly become everyone's punching bag.

Being a good Product Owner requires a solid understanding of your product, your customers and then making tough choices and "having balls". 

You won't please everyone. If your managers or internal stakeholders are an impediment to success, you need to bulldozer your way. 

If you can't or don't dare do that, Product Owner may not be the right job for you.