The introduction of Continuous Delivery systems and processes (when implemented correctly) brings four basic values to software development lifecycle:
- Structure
- Transparency
- Efficiency
- Quality
While all of those are very important, today I’d like to put some more focus on just one of them which is sometimes overlooked or not understood correctly – that is Transparency . In fact there’s another ‘T’ word which results from transparency –Trust, and I tend to think it’s the most important quality of a CD process.
Automated software delivery systems come to replace and improve the execution of multiple manual tasks involved in releasing quality software to end users. The benefits of automation are clear: create a repeatable process, eliminate human errors, replace expensive human time with cheaper machine time, streamline the tasks and there’s more…
But what we must never forget is that there are some perils too. Perils related to putting all the processes under the hood of our continuous delivery machine. When things are under the hood they go unnoticed, they don’t bother you… until they explode right into your face!
And here are a few examples of such hidden time bombs:
- Unchecked execution statuses.
- Dependency on local files and machine configurations.
- Missing cleanup procedures.
- Arbitrary sleep/wait periods.
According to my experience – all these are results of what I call “The Dream Process Syndrome”.
When building our CD pipeline we all have that Dream Process in our minds: a process that never fails, a process that can fix itself, a process we can rely on and pretty much forget about.
Of course it’s never that perfect – modern software development requires constant change. Tools, technologies, platforms – all are in constant flux and the CD pipeline has to be updated accordingly. But still – that craving for the Dream Process is always there! Don’t get me wrong – I think it’s a good thing – we should always strive for the impossible if we want to build the best possible. But sometimes this craving leads us to the opposite – instead of building the real Dream Process we start cutting corners in order to build the Dream Process illusion.
Here’s a good example of such corner cutting situation:
- Build A needs to mount a network disk B in order to copy build artefacts in post-build action
- Sometimes the mount command for disk B fails on the first, or even on the second time. (let’s say because of a faulty network connection)
- DevOps engineer adds a loop to retry the mount command 3 times before failing.
- Dream Process effect is achieved – everything works fine for a couple of months until….
- One day the mount command fails on the third time as well. The build process starts failing sporadically. The developers look at the log and can’t understand what suddenly went wrong. Builds aren’t getting deployed. DevOps folk aren’t available to fix this immediately so they tell the Devs: “simply rebuild, it should be ok next time..” or even worse – they pump the mount loop counter to 5, hoping that this will suffice.
- The Devs lose trust in the build process and opt for deploying the build they created manually on their machine.
- Do I need to continue?
Now – where did all this mess start? It started with the build engineer trying to hide the problem under the hood instead of exposing it in the open. In my book – the process should’ve been allowed to fail indefinitely until the network issues leading to failing mount are resolved once and for all. As the saying goes – ‘no pain, no gain’ – the real value of the CD process is manifested when it hurts. Yes, it should hurt everyone involved: you, the Devs, the Ops, the QA automation folks, the PMO. Because pain is the best motivator. So if you want to bring value – let your processes fail as often as possible.
You may ask: but who wants a process that fails all the time? How can you deliver software when it’s always failing? How can anyone rely on such a process?
You’re right of course – I’m exaggerating here. Your processes should only fail for the right reasons. False negatives are just as bad as false positives. Because they ruin the trust in your CD pipeline in exactly the same manner.
And that brings us back to the subject of this post. Your CD pipeline should be as trustworthy as possible. It’s not only the production line bringing the software to its customer-ready state. It should also strive to be the ultimate quality control gate that everyone involved can trust to sort out the unwanted garbage and to fail when the construction materials aren’t of the desired quality or when a cockroach slips in or a moment before the drive belt gets torn and the whole engine is blown to pieces.
You want the user of your system to trust it – and if you want trust – the following is a must :
- Check the exit status of every command you run.
- Fix the problems – don’t provide workarounds.
- Clean up your output – provide clear and understandable status and error messages.
- Communicate – make sure your users have at least the basic understanding of what’s going on under the hood. After all you’re dealing with technical people here and they usually like to know what they are doing.
- Test your processes – it’s kind of funny I have to mention this but I’ve seen too many CD process changes going into production without thorough testing.
The DevOps approach is all about breaking the walls between the silos, and you can’t break any walls without trust. Our CD pipeline should be the solid foundation for that trust and it’s our responsibility to build it that way – transparent and trustworthy as possible.
And leave the Dream Process where it belongs – in our vision of the perfect future.
Happy building!
originally posted on LinkedIn: https://www.linkedin.com/pulse/article/20140908143951-12769432-it-s-all-about-trust-from-continuous-integration-to-continuous-integrity?trk=prof-post