Hello, everyone! In this series of posts, I will cover a variety of topics that I think everyone who works with CircleCI (Circle) should know. For all the posts in this series, I’m assuming that the reader is familiar with the general concepts of CircleCI.
If you’re not, I suggest you read up on such concepts as workflow, job, step, command, and other basic terms. You could still follow along, but some things might not be clear right off the bat.
Battle-tested CircleCI tips and tricks (2 Part Series)
Part 1. Battle-tested CircleCI tips and tricks
Part 2. CircleCI – The Matrix
Circle CI config structure
First, we’ll talk about the structure of your config file (AKA the CircleCI YAML). It should look something like this:
Where each part of the Config file signifies the following:
- Config – This part will specify the version of your CircleCI config. Any Orbs, and their versions (I’ll talk about orbs later), and any parameters you wish to define “CI wide”
- Workflow – The main component of every Circle config, this is our entry point where we define workflows (there can be multiple; nightly, tests, etc). Each workflow is composed of multiple jobs.
- Jobs – A series of steps/commands that will be executed on a specified machine. You can write out your own steps, or use predefined commands.
- Commands – Those are steps that you have previously defined and will reference in your jobs. They are responsible for the actual work, they will have your shell commands that you wish to execute.
I have ordered them in this specific order to make it easier to read and understand what’s going on (you’re free to order them as you like).
The first thing you want to see is your configs for the run in order to see which orbs are present and what parameters are defined.
Next, the workflows. Your first level of abstraction. Just from the name you will see what their purpose is and what jobs they run.
Jobs are your second level of abstraction, they’ll have the actual commands, and you’ll be able to see what’s executed and in what order.
Lastly, the commands. There’s no abstraction here, this is the actual stuff that gets run (test, compile, build, push, etc).
Orbs
Orbs are another thing I want to talk about. They are:
“shareable packages of CircleCI configuration you can use to simplify your builds” – CircleCI.
I strongly suggest that before you start writing your own steps inside any job you look at what’s available online. Usually, someone already encountered the same problem you’re having and made things easier by writing an orb that you can use in your own config.
If you’re not sure what I mean by that, here’s an example: you want to build a Docker container and upload it to ECR (AWSs’ container registry). Instead of manually installing the AWS CLI on your machine and setting up your profile and writing all the commands yourself, you can use their orb that does all that for you! Amazing, I know.
There are many more use cases: Slack messaging, installing Node, Golang, etc. Go explore them yourselves.
Please note that they may not always fit exactly what you need! In some cases, it’s easier to write out the steps manually than use some limited functionality that an orb offers.
Secrets
Secrets! You will inevitably want to use some secret information that shouldn’t be written as plain text in your config file, this is where secrets come in. There are two types: organization, or project. Organization secrets are accessible from all projects/repos, while project-type secrets are only accessible from within the project/repo in which they are defined.
Their names are not actually secrets, but rather: Context, and Environment Variables, respectively. You can check out how to define contexts here, and env vars here.
Now, the question that comes to mind is when should I define a context secret, rather than an environment variable? Before I answer that, I want to clarify something about contexts, they are not exactly key-value pairs like environment variables are, but rather a collection of key-value pairs.
With this in mind, I can tell you my general rule of thumb for deciding which one to use. A collection of secrets, that serve the same purpose, or are going to be needed organization-wide, should be put in one context; e.g, a context named aws-access will hold: access keys, account ID, region, URLs, etc. While one-off, project-specific secrets should be placed in an environment variable.
Here’s a flowchart to get you started:
This will save you a lot of time if, for example, you change access keys. If you define them in a context, all you need to update is one or two fields, while if you define them as an environment variable, you will have to go from project to project updating those two values.
What’s next
This is the first post of the series which deals with more “philosophical” topics. Read the next posts to see more technical examples.
Cheers!
Part 2. CircleCI – The Matrix
Picture by Torn Banner Studios / Tripwire Interactive.
