War of the CI servers – GitLab vs. GitHub vs. Jenkins

Since their inception as little more than glorified shell scripts, continuous integration systems, or build servers as they were called, they have evolved from an exotic niche product to a central piece of any well-oiled software delivery project.

Along with the rise of agile methodologies and Continuous Delivery practices, the CI server has moved towards the center stage in your development lifecycle.

And while the market for CI servers is booming, it can be hard to tell the difference between them and choose which one to adopt. Should one choose Jenkins, GitLab CI, or maybe GitHub Actions? This blog post will guide you through the selection process and give you guidance for making the optimal choice.

The history of CI

Some 20 years ago, a ThoughtWorks employee named Matthew Foemmel built and released the first version of CruiseControl. While it probably wasn’t the first build system ever made, it stands as the first widely adopted and known one.

Looking at a screenshot of the original CruiseControl and the current GitLabCI, the major differences lie in the 20+ years of web UI development.

But surprisingly enough, not a lot of the essential information you need has changed in that time. Basically, it comes down to this:

• When did a build happen?
• Was it successful or not?
• Link to more information on the output of that particular build.

In essence, build pipelines are just that: a series of steps to validate if a given change in code results in a build releasable to the public.

The landscape: It is a wild west out there

The current landscape of CI systems is huge and booming. When I started in the DevOps world, the de facto standard was Jenkins, and the idea of build pipelines as code was something from the future. Everything was pointy pointy, clicky clicky UI work.

Nowadays, we have more than a dozen different offerings, all competing for market share. 

They all have their unique features and purported benefits, making it hard to figure out if you miss any crucial features by opting for CI Brand X  over Brand Y.

When you search on the web, software providers usually show their made-to-measure feature comparisons vis-a-vis the competitors, invariably showing their superiority over the challengers. 

I will focus on three offerings in the following sections: Jenkins, GitHub Actions, and GitLab CI. I picked these three products because they are the ones we see most often in the wild at the moment, and they represent the mainstream of offerings.

Differentiating between the contenders

All three vendors have their strengths and weaknesses and depending on your needs and preferences, some of them matter more than others.

The matrix below lists what I value as the most important factors when choosing a CI system.

* DAG will be explained later in the post.

As you can see from the sheet, the field is pretty even. Design choices lead some to pick YAML instead of a DSL, or to prefer more built-in features instead of having everything as external tools like plugins or Actions as GitHub likes to call them.

Jenkins, however, starts to seem more and more dated. For instance, it doesn’t have any starter templates that enable you to be up and running with “Hello Gradle” or the like in minutes. 

Jenkins also lacks the capability for directed acyclic graphs (DAG) for pipeline orchestration, which can lead to longer feedback times, which we actively discourage.

A bit about DAG Pipelines

When building pipelines, you typically want to run as many things as possible in parallel. But not all jobs take the same amount of time. If the CI system only supports fan in/out, all jobs in a given “stage” need to complete for the next batch of jobs to run. 

With DAG (Directed Acyclic Graph) pipelines, you can specify when any of your jobs can run (what other jobs should be finished beforehand). The CI system then figures out the order accordingly. This is very important, especially for big multi-tier projects and/or mono repos.

Who wins the war?

Here comes an opinionated part of the blog: unless you feel that a unique feature of a particular vendor is a critical differentiating factor for you, picking any of the CI systems enables you to release quality software (given that you have written quality software, to begin with).

One such feature in the past was CircleCI offering both OS X and Android support as one of the first SaaS solutions. This made CircleCI ideal for mobile app developers who did not want to buy a Mac for building IOS apps.

And unless you think that hosting your CI operation (servers instances for the main server and build agents) is part of your competitive advantage, think hard and long about if there ain’t someone (vendor or third party) that can do this for you. Then you can spend your hard-earned innovation tokens elsewhere.

So, why are so many companies making the same thing?

Because there is a war going on for the entire development ecosystem.

From the vendor standpoint, a customer using several of their tools for issue management, code repository, build server, etc. has a much harder time dumping them because things tend to “glue” together. (From a vendor perspective, vendor lock is actually a positive thing.)

The six steps to wisdom

Below are some practical steps that I have found helpful when choosing (or untangling) your CI system:

• Assess your environment. If you are already faring well with Github/Gitlab/Atlassian, stick with their build system, and leverage the synergies from the vendor. Jenkins is harder from a maintenance perspective, and their plugin jungle makes it messy to upgrade. Furthermore, being detached from the source code repository server forces you to have one more app(™) to manage (or be managed by).
• Harden your delivery chain. In recent years supply chain attacks have been one of the scariest methods to poison software from inside the delivery chain. Security features like two-factor authentication, different ways of verifying who pushed code (e.g. code signing), easy role-based access control (RBAC), software library or source code vulnerability scanning, etc. are worth gold here. GitHub and Gitlab win big since in Jenkins you not only have to roll most of this on your own/buy separate tools, but Jenkins and the plugins are a constant source of CVEs.
• Review your pipelines. If you are using many vendor-specific tools and plugins, the chances are that you will not be able to make a switch easily.
• Reproducibility anywhere. Ideally, you want to be able to release, even if your CI system is down. By wrapping most of your build work inside scripts, you make it easier to test each step on a developer machine and create emergency releases in case your CI system is down, and you need to release.
• Containerize your workloads and delivery. This has many added advantages other than just making your pipeline reproducible. If you are still struggling with “but it works on my machine”, containers are your go-to solution.
• Remember to keep your pipeline dumb. If you have logic in your pipeline that needs to define your success or failure criteria, your pipeline is too intelligent. Take a step back and ask yourself, “Is this really necessary”? One way to improve your pipeline is to organize a workshop for your team around the pipeline using the Pipeline game. This way, you can see beyond the nitty-gritty of details and focus on the bigger questions like what quality gates you need and how to determine you are safe to release.

Seeing is believing: the Eficode example pipeline

Finally, if you would like to see the differences in how the syntax from different vendors looks, we have created the same pipeline with all the CI systems discussed above (and a few more).

The pipeline looks something roughly like this in all the systems:

Steps:

• Clone down: makes the git clone, and prepares the repo for being distributed to the parallel steps
• Test: runs the gradle test command found in ci/unit-test-app.sh
• Build: runs the gradle build command found in ci/build-app.sh
• Build Docker: runs a linting of the dockerfile, building of the docker image, and pushes it up to the hub
• System tests: runs two tests, first a k6 performance test with docker-compose, and after that a docker-compose file with a python test to test the application.

You can look more deeply into the differences and similarities between the syntaxes on the Github repository: https://github.com/eficode-academy/war-ci-servers