encodeering

Creating an armhf image on a different hardware platform requires five to ten times more patience than creating the same image on a native platform. A few months ago I've started with Travis CI to build the official docker images for armhf and very few images always exceeded to maximum build time of 50 minutes, due to some costly compilation steps. 

So in order to build those images as well, I had to experiment quite a little and eventually came across the following document speeding up your build, which recommends a parallelizing build matrix to get things done faster. In fact, each build step is getting a full lifetime of 50 minutes, which I thought could be a promising option. 

There are a couple of boundary conditions for this approach, but in short, each build step uses the result of a preceding build step, if it's not the first to stop the recursion. This can be achieved using a different set of environment variables to control the flow of the build script or scripts. In order to get this work, one has to at least

• think about how to correctly share preceding artifacts with other independent build steps - remote file caches, docker registry, ...
• split the build into phases, that fit the project and build tool requirement - which may not be that desired or even that easy
• set the limit of concurrent job to one - building each step one after the other, increasing the overall build time
• set a configuration for allow_failures and fast_finish to handle error situations - don't build a subsequent step with an incorrect artifact

This is a lot to take care of and I really would recommend this approach only, if there are no other options left. But for the moment, it pushed everything further, until, yes, things went worse again :) and one of the build steps started exceeding the 50 minutes limit again - due to some more compilations steps of course. 
Unfortunately, that step couldn't be split reasonably anymore. A newer version of the emulation software was required anyway, so different compilation options have been used to make it run faster, even knowing that it would only alleviate the symptoms. Luckily, there was no success.

A spreadsheet about CI tools reveals quite a lot of candidates, but don't get me wrong now, I'm very happy with Travis, it's just about those few projects, that require some special attention. Travis legacy infrastructure (requiring sudo) is subject of change and hopefully, they spend a multicore environment to open source projects as well.

Anyhow, you should read the spreadsheet carefully, if you'd like to know more about the pros and cons, as they won't be repeated here.

The journey started with Circle CI. A free plan of 1500 build minutes per month was quite different to other tools with a maximum of build minutes per job, but unlimited build counts. Sure, nightly builds for a couple of projects aren't possible with that plan, but a rollback of the mentioned split should be doable. It turned out that a cross-compilation feature isn't supported yet.

The route then led to Semaphore CI, offering a free plan for open source projects with 60 build minutes this time, which is only 10 minutes more than Travis, but backed with a multi-core infrastructure instead, making a huge difference as we will see. A complete non-splitted build with all configuration and compilation steps varies between 30 to 40 minutes and I assume, that make utilizes the hardware resources more efficiently - the build seems to be CPU bound here. A simplified build-matrix can be constructed using the so called Thread capabilities, which is fine for most situations and two Threads are offered by the plan, each providing the full amount of time as far as I can see. One drawback - at least to me (single source of truth) - is that Semaphore doesn't use an SCM configuration file, but a web interface instead, to configure the build jobs. This can be seen differently, as it simplifies some configurational aspects on the other hand.

After the successful test, there was one more platform on the list, called Shippable. They offer a multi-core environment as well and promote themselves as being Travis compatible, which might compensate for the mentioned drawback. Unfortunately, they have the same limitation and don't provide a cross-compilation feature yet, if you don't bring your own device to be exact here. Indeed a nice feature to support hybrid configurations.

At the end of the short journey, I can say, that the landscape of available tools is amazingly huge and each of the tools has quite its own strengths.
Sometimes its easier to find the information in the documentation and sometimes its easier to use a sandbox project to find the tool that suits your project most. For the latter, I can say, that the decision to use bash scripts for the build and yaml configurations for the infrastructure, made it possible to switch the platform more easily. The decision was made for debugging purposes originally, running a build on a local machine, but reflecting once again, separating the proprietary configuration for infrastructure from the rest was the reason for an affordable switch between the CI platforms.

The photograph for this article was taken by Nicole Kasper.