One can not talk about modern software delivery without mentioning Infrastructure As Code (IAC). It’s one of the cornerstones of DevOps. It turns ops into part-time coders and devs into part-time ops. IAC is undoubtedly a powerful concept – it has enabled the shift to giant-scale data centers, clouds and has made a lot of lives easier. Numerous tools (generally referred to as DevOps tools) have appeared in the last decade to allow codified infrastructures. And even tools that originally relied on a user-friendly GUI (and probably owe much of their success to the GUI) are now putting more emphasis on switching to codified flows (I am talking about Jenkins 2.0 of course, with it’s enhanced support of delivery pipeline-as-code).
IAC is easy to explain and has its clear benefits:
- It allows automation of manual tasks (and thus cost reduction)
- Brings speed of execution
- Allows version control for infrastructure configuration
- Reduces human error
- Brings devs and ops closer together by giving them a common language
But why am I writing about this now? What made me revisit this already quite widely known and accepted pattern? ( Even many enterprise organizations are now ready to codify their infrastructure )
The reason is that I recently got acquainted with an interesting man who has a mind-stimulating agenda. Peter Muryshkin of SIGSPL.org has started a somewhat controversial discussion regarding the future of devops in the light of overall business digitalisation. One thing he rightfully notices is that software engineering has been learning a lot from industrial engineering – automating production lines, copying Toyotism and theory of constraints, containerising goods and services, etc. The observation isn’t new – to quote Fred Brooks as quoted by Peter :
“Techniques proven and routine in other engineering disciplines are considered radical innovations in software engineering.”
This is certainly true for labour automation that has existed long before IAC has brought its benefits to software delivery. It’s also true for monitoring and control systems which have been used on factories since the dawn of the 20th century and which computers started being used for in the 1960-ies.
But the progress of software delivery disciplines wasn’t incremental and linear. The cloud and virtualization virtually exploded on us. We didn’t have the time to continue slowly adapting the known engineering patterns when the number of our servers suddenly rocketed from dozens to thousands.
In a way – that’s what brought IAC on. There were (and still are) numerous non-IAC visual infrastructure automation tools in the industry. But their vendors couldn’t quite predict the needed scale and speed of operation caused by the black hole of data gravity. So the quick and smart solution of infrastructure-as-code was born.
And that brings us to what I’ve been recently thinking quite a lot about – missing visualization. Visibility and transparency (or measurement and sharing) are written all over the DevOps banners. Classic view of IAC actually insists that “Tools that utilize IaC bring visibility to the state and configuration of servers and ultimately provide the visibility to users within the enterprise“ In theory – that is correct. Everybody has access to configuration files, developers can use their existing SCM skills to make some sense of system changes over time… But that’s only in theory.
The practice is that with time the amount of IAC code lines grows in volume and complexity. As with any programming project – ugly hacks get introduced, logic bugs get buried under the pile of object and component interactions. (And just think of the power of a bug that’s been replicated to a thousand servers) Pretty soon only the people who support the infra code are able to really understand why and what configuration gets applied to which machine.
In the past years I’ve talked to enough desperate engineers trying to decipher puppet manifests or chef cookbooks written by their ops colleagues. Which makes me ask if maybe IAC sometimes hinders devops instead of enabling it…
Even the YAML-based configurations like those provided by Ansible or SaltStack become very hard to read and analyze beyond simple models.
As always is the problem with code – it’s written by humans for machines, not for other humans.
But on the other hand – machines are becoming ever better at visualizing code so humans can understand them. So is that happening in the IAC realm?
In my overview of Weapons of Mass Configuration I specifically looked at the GUI options for each of the 4 ninja turtles of the IAC and sadly found out that not even one of them got any serious praise from the users for their visualization features. Moreover the GUI solutions were disregarded as “just something the OSS vendors are trying to make a buck from”.
I certainly see this as a sign of infrastructure automation still being in its artisanal state. Made by and for the skilled craft workers who prefer to write infra code by hand. But exactly as the artisans had to make way for the factories and labour automation of the industrial revolution – a change is imminent in the current state of IAC. It’s just that the usable visualization is still lacking, the tools still require too much special skills and the artisans of the IAC want to stay in control.
Don’t get me wrong – I’m not saying our infra shouldn’t be codified. Creating repeatable, automated infrastructure has been my bread and butter for quite some time and tools like Puppet and Ansible have made this a much easier and cleaner task. I just feel we still have a long way to go. ( Immutable infrastructure and containerisation while being a great pattern and having benefits of its own also relies heavily on manual codification of both the image definitions and the management layers. )
Infrastructure management and automation is still too much of an issue and still requires too much special knowledge to be effectively handled with the existing tools. Ansible is a step in the direction of simplicity, but it’s a baby step. Composing infrastructure shouldn’t be more complicated than assembling an Ikea bookshelf – and for that new, simpler, ergonomic UIs need to be created.
Large-scale problems need industrial level solutions. Let’s just wait and see which software vendor provides such a solution first – one which will make even the artisans say ‘Wow!’ and let go of their chisel.
And with that said – I’ll go back to play with my newly installed Jenkins 2.0 instance for some artisanal pipeline-as-code goodness. 🙂