The story of iOS CI at Appunite (Part 1)

It's almost the end of 2019 - exactly 3 years after the U.S. presidential election and maybe America isn't made great again but we all need to admit that it was a pretty good electoral slogan. A company that I am working in used what made Ronald Reagan’s and Donald Trump’s campaigns successful and made its own motto: - "Appunite - Make software great again". What if you ask me what makes us create such high quality software? I would answer that it’s mostly people, but still "mostly" and not "only" because apart from hard work, there are always tools that help to achieve it. One of such tools is Continuous Integration system that we are using at Appunite to test and deploy our software every day, hour and minute. Today I would like to tell you a little story about CI infrastructure that I've been working on, which is used for building iOS applications.

Quick recap of what CI indeed is

When you think about CI, you probably imagine some magical system that schedules to run build on some physical computers. At Appunite, we use Gitlab CI, which is a natural way of continuous integration when using Gitlab as a source control tool. To make it clear, let me show you the simplest possible Gitlab CI setup:

As you can see, there are 3 components required:

• the obvious one - developer - if there is no developer, there is no need for CI
• Gitlab instance - used to coordinate all work between all projects and runners (instances that are responsible for building)
• CI machine / runner - which is communicating with the main Gitlab coordinator and getting jobs to build on the local computer

Our setup for 2017

Back in 2017, when I joined the company, the CI setup looked more or less like this:

Our CI fleet consisted of 6 Mac Mini's. Each of those machines had some macOS installed, a specific Xcode version (we are a software house and build many projects under different Xcode versions), and of course a software that communicates between that machine and the main Gitlab service. This software is represented by Go language mascot because it is written in Go. If you are interested in reviewing its source code, you should visit gitlab runner repo.

While this setup looks pretty simple - we can build concurrently on 6 different computers with different Xcode versions - there is actually nothing more positive than I can say about it. As you may guess - we soon started to suffer from many different problems that tend to interfere with our every-day work:

• Sometimes new Xcode required new macOS - when it happens, we need to put Mac Mini on the desk and update its software (takes around 2 hours for a single computer)
• Sometimes, for a particular project, it successfully builds on machine A but fails on B - this is not a trivial issue, but the same machines (same Xcode version, same macOS) are still physically different computers, they may behave differently depending on some other software installed
• Similar problem to the one above, which I used to call "It was working yesterday" - Sometimes, some changes (even those made by different project) may result in not building successfully on the same machine that it was building a few days ago
• What if we need to install a new Xcode or some version is broken? - unfortunately, we need to put a computer on the desk and manually install software on it (more than 1 hour for a single computer)

Well, having such problems caused, we wasted a lot of time with every Xcode release and spent dozens of hours debugging every user issue.

New era

Our previous setup was flexible enough to allow us to unplug one computer and start experimenting with it. We came up with something completely different - we started virtualizing macOS with Xcode. A single machine instance looked like this:

Now, gitlab-runner is not running CI jobs directly on mac, but doing this through virtualization software (Virtualbox in this case). With this setup, we have achieved several new benefits:

• Multiple Xcodes on one device - for now, we can install as many Xcode instances as the machine's disk space will allow,
• Backups - for every created image, we can backup it and, in case of some issues, restore from such a "safe state"
• Templates - every run of a particular image can have the same entry point and revert all the mutations done by the previous run
• We don't need 6 computers! - since we can have several Xcode versions per single computer, we were good enough with reducing the number of instances by 50%
• Sharing images - when new Xcode is released, we still need to configure it manually, but then, we can distribute a prepared image across all computers with just copy-paste of the image

At the same time, we solved most of the problems from the previous architecture. We don't need to update macOS on computer but just on a virtualized instance and thanks to templates and backups and copying of images, we have no problems of unstable runs between days/computers. This saved us a lot of time after all and simplified the process of adding new versions of Xcode.

But when everything sounds so ideal, then, of course, new issues are on the way for us to face. What were the new issues that we faced after introducing the improvements above? This is what we’ll look into in the next article…