Breaking build is not a crime
The team must simply keep the build working at all times. If the build fails, it should be a “stop the presses” event and the team should meet to quickly resolve the issue.and later in that section:
I have every developer run the continuous-build script before they commit.Final quote:
They (CI tests) should never fail. If they fail, then the whole team should stop what they are doing and focus on getting the broken tests to pass again. A broken build [...] should be viewed as an emergency[...]In another wonderful book Continuous Delivery by Jez Humble and David Farley the authors go way further. They present 7-point plan that we should follow on every commit (!):
They suggest the following best practices:3.Run the build script and tests on your development machine to make sure that everything still works correctly on your computer
[...]
5.Wait for your CI tool to run the build with your changes.
[...]
7.If the build passes, rejoice and move on to your next task.
[...]
If the commit succeeds, the developers are then, and only then, free to move on to their next task.
To summarize what literature says about continuous integration:Never Go Home on a Broken Build
[...] When the build breaks on check-in, try to fix it for ten minutes.
- always run all the tests before committing to make sure everything is green
- look closely at your CI builds to make sure they pass, don't proceed with further tasks
- if you break the build, you must treat this as an emergency and fix it ASAP
- you have very little time to fix, otherwise revert your changes
I fully understand this point of view and behaviour, but this doesn't mean I agree. I feel this workflow is just plain wrong. I am aware that the whole team is working on the same HEAD/trunk in version control so breaking it is possibly a show-stopper for all of them. But I'm against treating CI/source control as some scarce resource that is so mission-critical.
Continuous integration server and VCS should be your personal team-mate, doing work for me. No one is really paying me for staring for, say, 5 minutes at my IDE before each commit to make sure all tests still pass. If that's the case, I'm suppose to watch CI build blindly for next 5 minutes. If I broke the build, they expect me to drop everything and just jump in, trying to fix build within 10 minutes, as Continuous Delivery suggests. All this in emergency, stressful manner. Why?
Back in the days of Java 1.4 we were taught that concurrent programming using
wait() and notify() is hard. But instead of giving up concurrency we came up with better and easier to use abstractions and libraries. At the same time we were reluctant to rename classes as this also renamed .java files, operation not well supported in CVS. But instead of keeping old names forever we migrated to superior SVN. Now because of technical limitations of continuous integration servers and VCS we should treat CI server as a very expensive über-assertion that should never fail. Technology deficiencies seem to affect our productivity and workflow.I want to break the build whenever I want! When I'm done and my new tests pass, I just want to commit/push my new stuff and let CI server perform full testing. I hope everything flies but if not, I don't want to feel guilty. I don't want to stay late or apologize my team-mates. I will fix these unexpected problems when I can. It's not production, it's just my experimental new feature failure that no one cares.
This naturally leads to an idea of feature branches. The concept is simple: you develop your feature on a separate branch, CI server might even build all your changes and when you feel you are ready, you simply merge your changes back to mainline. The problem with feature branches is that it's no longer continuous integration. After days of development your feature might be green and ready alone, but merging it back might be extremely problematic. Also other team members might benefit from your changes, even if they are not complete (but already green). All these observations led me to the following requirements:
- I want to push my changes as often as I need
- CI server should build my changes in isolation so that if they break, no one sees them or cares
- if my changes are good, I want them to be automatically and immediately visible to others
- I also want to see changes made by others as soon as possible
master. The first step is to create a separate branch and commit to that branch. We should never commit to master. When I think I'm ready with my new feature I simply push that branch and move on. No running tests locally, no nervous monitoring of CI server. Just push and approach new challenges. This may lead to great savings in time if your test suite takes few minutes to run.First things first, here is how you set up Bamboo. Under Plan Configuration and Branches select the following highlighted options:
Automatically manage branches will discover and build all new branches automatically. Branch Merging Enabled allows Bamboo to automatically merge new branches with
master one way or another. In the Gatekeeper configuration we tick Push on option. Here is how it works: I make several commits to my feature branch. You can push them immediately or after some time: | * 894217c (HEAD, feature, origin/feature) More tests |
feature branch is placed on top of master. master branch is still green and my, possibly breaking, changes are isolated. Here comes the magic. I configured Bamboo to discover new branches and build them automatically:
What's so special about that? Bamboo tells me that my changes are fine so I am free to integrate them into the mainline (
master branch). Am I? No, Bamboo did it already! It built my changes, found them to be green and automatically merged them into `master so that others can see them. Merging was simple, it's just fast-forward: | * 894217c (HEAD, origin/master, origin/feature, master, feature) More tests |
OK, was it really that interesting? After all we would get the same result by simply pushing directly to
master... Well, but what happens if we are pushing breaking changes to unmodified mainline? This is the terrifying moment in most of the teams. I just pushed breaking changes and everyone is yelling at me. Fix. Fast. Y U NO RUN TEST? But not in this approach: | * cc4ea63 (HEAD, origin/feature, feature) Experimental changes |
Here is where all this pain starts to pay off:
feature branch might be broken, but master is untouched. No merging occurred. Only my very own, private branch is broken. Other developers are unaffected. If this build was green, my experimental changes would have been automatically merged to master and pushed. But I made a mistake and they remain hidden. No one cares, my team mates still see stable master. I can go for lunch or fix it tomorrow. No stress, no peer-pressure. When I get it right, Bamboo will automatically apply my fix. This was all very easy as Bamboo could use fast-forwarding instead of ordinary merge. But what if we try pushing good changes to modified remote
origin/master? Suppose we are working on our feature but in the meantime other developer pushed some changes to bugfix branch which happened to be correct so Bamboo decided to merge them immediately: | * d600da5 (HEAD, feature) Cosmetics |
feature branch was not yet pushed to main repository while bugfix branch is already integrated. How will Bamboo deal with feature branch being pushed? The behaviour is a bit more complex, but still manageable: Bamboo first checks out master (including bugfix branch already merged) and tries to merge changes from feature branch. If merge was successful (no conflicts), ordinary build is performed. If build is successful, merge results are pushed to master: | * e58a2db (origin/master) [bamboo] Automated branch merge |
feature branch changes into master branch. This approach works but has several drawbacks:- after a while your
masterbranch history might consist of barely Bamboo generated commits. I'd rather see ordinary commits there - automatic merging in Bamboo might fail
- my
featurebranch still doesn't havebugfixbranch changes already merged into mainline
feature branch first locally with master and push that. In this scenario you are almost guaranteed that remote merge on Bamboo will never fail (only fast forward), it's predictable and you work on the latest master state. And BTW wondering what would happen if automatic merging on Bamboo fails?
Summary
This approach for working with version control brings best of both worlds: feature branches and continuous integration. Because each developer is working on a separate branch (or even repository!), broken commit won't ever make it tomaster branch/mainline. On the other hand automatic merging will make sure our feature branch is always up-to-date and we won't run into issues when trying to merge days worth of work. Moreover good commits are immediately visible to others while bad ones remain hidden.
Tags: bamboo, continuous integration, git