Dynamically spinning up Jenkins slaves on Docker clusters

Introduction:

Being able to dynamically spin up slave containers is great. But if we want to support significant build volumes we need more than a few Docker hosts. Defining a separate Docker cloud instance for each new host is definitely not something we want to do – especially as we’d need to redefine the slave templates for each new host. A much nicer solution is combining our Docker hosts into a cluster managed by a so-called container orchestrator (or scheduler) and then define that whole cluster as one cloud instance in Jenkins.
This way we can easily expand the cluster by adding new nodes into it without needing to update anything in Jenkins configuration.

There are 4 leading container orchestration platforms today and they are:

Kubernetes (open-sourced and maintained by Google)

Docker Swarm (from Docker Inc. – the company behind Docker)

Marathon (a part of the Mesos project)

Nomad (from Hashicorp)

A container orchestrator (or scheduler) is a software tool for the deployment and management of OS containers across a cluster of computers (physical or virtual). Besides running and auditing the containers, orchestrators provide such features as software-defined network routing, service discovery and load-balancing, secret management and more.

There are dedicated Jenkins plugins for Kubernetes and Nomad using the Cloud extension point. Which means they both provide the same ability of spinning up slaves on demand. But instead of doing it on a single Docker host they talk to the Kubernetes or Nomad master API respectively in order to provision slave containers somewhere in the cluster.

Nomad

Nomad plugin was originally developed by Ivo Verberk and further enhanced by yours truly while doing an exploratory project for Taboola. A detailed post describing our experience will be up on Taboola engineering blog sometime next month.
Describing Nomad usage is out of the scope of this book, but in general – exactly as the YAD plugin – it allows one to define a Nomad cloud and a number of slave templates. You can also define the resource requirements for each template so Nomad will only send your slaves to nodes that can provide the necessary amount of resources.
Currently there are no dedicated Pipeline support features in the Nomad plugin.
Here’s a screenshot of Nomad slave template configuration:

Kubernetes

The Kubernetes Plugin was developed and is still being maintained by Carlos Sanchez. The special thing about Kubernetes is that its basic deployment unit is a kubernetes pod which could consist of one or more containers. So here you get to define pod templates. Each pod template can hold multiple container templates. This is definitely great when we want predefined testing resources to be provisioned in a Kubernetes cluster as a part of the build.

The Kubernetes plugin has strong support for Jenkins pipelines with things like this available:

podTemplate(label: 'mypod', containers: [
 containerTemplate(name: 'maven', image: 'maven:3.3.9-jdk-8-alpine', ttyEnabled: true, command: 'cat'),
 containerTemplate(name: 'golang', image: 'golang:1.8.0', ttyEnabled: true, command: 'cat')]) 
{
  node('mypod') {
    stage('Checkout JAVA') {
    git 'https://github.com/jenkinsci/kubernetes-plugin.git'
    container('maven') {
      stage('Build Maven') {
        sh 'mvn -B clean install'
      }
    }
   }
   stage('Checkout Go') {
     git url: 'https://github.com/hashicorp/terraform.git'
     container('golang') {
       stage('Build Go) {
         sh """
           mkdir -p /go/src/github.com/hashicorp
           ln -s `pwd` /go/src/github.com/hashicorp/terraform
           cd /go/src/github.com/hashicorp/terraform && make core-dev
         """
       }
    }
  }
}

There’s detailed documentation in plugin’s README.md on github.

Marathon

There is a Jenkins marathon plugin but instead of spinning up build slaves it simply provides support for deploying applications to a Marathon-managed cluster
It requires a Marathon .json file to be present in the project workspace.
There’s also support for Pipeline code. Here’s an example of its usage:

 marathon(
   url: 'http://otomato-marathon',
   id: 'otoid',
   docker: otomato/oto-trigger')

Docker Swarm

I used to think there was no dedicated plugin for Swarm but then I found this. As declared in the README.md this plugin doesn’t use the Jenkins cloud API. Even though it does connect into it for label-based slave startup. This non-standard approach is probably the reason why the plugin isn’t hosted on the official Jenkins plugin repository.
The last commit on the github repository dates back 9 months ago, so it may also be outdated – as Docker and Swarm are changing all the time and so does the API.

Learn More:

This post is a chapter from the ebook ‘Docker, Jenkins, Docker’ which we recently released at Jenkins User Conference TLV 2017.  Follow this link to download the full ebook: https://otomato.link/go/docker-jenkins-docker-download-the-ebook-2/