Successful Continuous Integration

Continuous Integration is a staple in most organizations these days. Setting up CI is seemingly trivial, given Web instructions and a bit of hardware, any developer can do that in a reasonably short period. However, CI in many organizations is highly dysfunctional, sometimes nobody is aware of the dysfunction - sometimes, it is visible, but the root cause is just out of reach.

Let's take a deeper look at CI:

CI is much more than just a tool!

 Too often, people jump into the solution space too quickly and say "We need a CI tool". That's like saying "We need a car" without even considering where you intend to go: A tool is not a solution unless you define the problem first. For many developers, it's simply a habit, because every good software project these days has CI. But why? Let us start differently.

Process

Let's ignore for a second the seemingly trivial "After a merge, automatically build and run tests, then report failures" aspects of the CI process and dig into the real question "What do we actually expect from our CI process?" - what do you expect from your CI process? That should be, first and foremost, "instant feedback". But: Feedback on what? Based on what feedback you are looking for, you must design your own CI process to give you the most important feedback clearly, quickly and reliably. So, the difficult question becomes "How do we get that?"

Scope

In order to answer the question, "What feedback are we looking for?", we first need to answer the question "What does Integration actually mean?"
Are we trying to integrate:

• pieces of code into a repository? 
• features into a system?
• activities into processes?
• processes into process chains?
• components into a larger system?
• systems into a larger ecosystem?

Depending on what your question is, the feedback you are looking for differs.
Just as an example, let us look at the question "Is our User Account functionality OK?" in comparison to the question "When 10000 users are on our site at the same time, will everyone be served without encountering problems?" - you obviously need a completely different approach to get feedback here.
Before you can answer how you want to get the result, you must define what result you expect and why you need that information.

Test Suite

Once you know what you want and how you want to get it, that obviously just needs to get done. However, it shouldn't just "get done", but it should get done well. Common problems in CI test suites are:

• Poor coverage, i.e. the information "it passed" is not meaningful
• Flakiness, i.e. the information "it failed" is not meaningful
• Interdependence, i.e. the information "27 tests failed" is not meaningful
• Slow tests, i.e. the information "It finished" is no longer meaningful

Associated with these problems are low value of feedback, high maintenance efforts and productive defects - visible in dissatisfied customers and high cost of development.
Frequently asking "Does our test suite solve the problem we have?" is quite important. While adding tests as needed is easy, both weeding out unnecessary tests and refactoring for optimal information value are often tough calls.

Mindset

The biggest problem with CI in many organizations is that they do not realize that CI is neither a tool nor a process - but a way of working supported by these. Developers do not serve the CI tool and their job is not to keep the master green. As long as developers consider CI a burden rather than a help, then no specific implementation will yield the desirable benefit. Much rather, CI helps developers get their job done well and a red master is an indicator that something is wrong.
Good CI will not only fix a Red Master, but trigger Continuous Improvement [also "CI"] when the Master goes Red: "Why did it turn Red? What can we do to avoid that situation in the future?"
Working on a proper CI mindset goes a long way in successfully implementing CI. A key aspect of this mindset is frequently asking "Which problem are we actually trying to solve with CI? Do we do that?"

Tool

A fool with a tool is still a fool: The tool must be adequate to solve the problem at hand and set up properly to deliver the intended benefit.
Initially, CI tools are empty shells that trigger build jobs and report Pass or Fail. Configuring a source code repo and defining the test suite to execute is done quickly. But you may want to set up individual build chains for different problems, depending on what you are trying to achive. What these chains do, how they interact and what you do with the resulting information is something the tool can not help you with.

Summary

CI supports good software development, but it is not a mandatory component of software development. If your current CI is limited to having an automated build and test job but you've never considered the big picture, chances are you are doing Cargo Cult CI - lots of work for little value. When you see symptoms like developers ignoring Red Master or not even waiting for the CI to complete, you should definitely take a break and do a Retro with the topic: "Let's talk about CI."