Please - the Thought Machine Build System



By Peter Ebden, Chief Infrastructure Officer

December, 2017

One of the crucial requirements for any software project is the question of how it gets built. There are many options out there, but today I’m going to explain why Thought Machine decided to build our own system to solve this problem.

Please when it’s running

To be clear, when we talk about a build system, we’re referring to the level above the compiler - so not gcc itself, the system that works out what to build and when, moves files around and instructs gcc about what to do. We haven’t written our own compiler for anything (yet).

Our development takes place in a single repository of heterogeneous code - a.k.a. a monorepo. Several of us had worked in similar environments before and were keen to replicate the same benefits (or possibly avoid disadvantages we’d seen elsewhere). I’ll leave that there since this topic has been discussed at great depth elsewhere, but suffice to say we’re sold on monorepos. Our repo contains over 500,000 lines of code, spread across various languages; in roughly decreasing order of size:

  • Python
  • Javascript
  • Go
  • Java
  • YAML (NOTE: Primarily for Kubernetes manifests - hence it’s as much in need of a build process as other languages.)
  • Protocol Buffers
  • Other Stuff (NOTE: Sass, C++, SQL, Thrax grammars, HTML, Bash scripts...)

All of these have different requirements for their build processes, but we were adamant from the beginning that we did not want to cobble together a Frankenstein’s monster that required knowing different tools to interact with different parts of our repository. That’s annoying in a practical sense but also bad for overall team cohesion - it promotes siloing because there’s a higher barrier to entry if you’re not familiar with the tools to work on a language. There’s also a lot of interaction between the different languages - for example it’s typical to use protobufs / gRPC to generate service stubs which are compiled with the real application code, the final binary built into a Docker image, and that image referenced from a Kubernetes manifest. It’s very important for us that the experience there is seamless - when changing a .proto file, developers should never need to remember to update generated code anywhere, and certainly not check it in; the changes should flow through everything downstream from it automatically.

A bit of history

In pursuit of this goal, we had an initial flirtation with Gradle, but ultimately found it unsatisfying. A few of us had fond memories of Blaze at Google and went on the hunt for something similar; we settled on Buck which kept us going a bit longer, but eventually we found it tricky to extend to some of what we wanted to do with it - but also we hadn’t found another option that would clearly be better out of the box. At that point a high-level design meeting was held (NOTE: At a pub one evening, which is a traditional site for innovation.) and we decided to have a crack at doing it ourselves (phrases like "how hard can it be" were uttered).

Along with the implicit goals of being able to build our stuff, there were a few things we were aiming for specifically with Please:

  • Fast (of course)
  • Lightweight (no daemons, no system-level dependencies)
  • Easily extensible
  • Powerful query capabilities (to allow running minimal sets of tests on changes)

Reproducibility and hermeticity are also very important goals - since we’re building software for the finance industry, we need to be able to demonstrate a high level of care about what we’re consuming as build inputs. Adding arbitrary third-party dependencies is something we need to be careful about; there have been plenty of examples of ecosystems being compromised through dependencies, so we’re willing to take a bit more time to be sure what we’re getting.

After a few months of hacking (NOTE: Keeping it secret from the CEO during that time was a challenge), we had a v1 that was self-hosting and able to fully build our entire repository. At that point we were able to switch all our developers over, and then start adding features in earnest. This was around mid-2015 and so we’ve had time for quite a lot of those - for example we’ve subsequently added distributed caching, cross-compiling, persistent workers, test containerisation and sandboxing.

How it Works

As mentioned earlier, Please is very strongly inspired by Blaze, and has similar BUILD files that contain a high-level description of what to build & how. These files are written in a subset of Python so it’s possible to program them arbitrarily, but in most cases they take a fairly declarative form. For example:

go_library(
    name = 'cli',
    srcs = glob(['*.go'], exclude = ['*_test.go']),
    deps = [
        '//third_party/go:go-flags',
        '//third_party/go:terminal',
    ],
    visibility = ['PUBLIC'],
)

go_test(
    name = 'flags_test',
    srcs = ['flags_test.go'],
    deps = [
        ':cli',
        '//third_party/go:testify',
    ],
)

This example defines a reusable Go library, and a unit test on it. Both the test and the library declare their source files, and refer to other build targets that they depend on. This explicit manifest of inputs for each rule is a little work for developers to write out, but it allows the system to be precise about correctness at every step. Crucially, it means that when something changes Please can work out the exact set of targets to rebuild or tests that need to run; it can avoid rebuilding things that it doesn’t need to, while also being sure that there aren’t any stale outputs which could lead to incorrect results.

It’s also extensible - since the build language is programmable, all the rules for how to compile various languages are written in it. Please ships with support for C, C++, Go, Python, Java and shell rules - but those aren’t treated specially by the system in any way and it’s possible to write & load more of your own dynamically. That gives a clean separation of concerns so the core system can focus on the things it’s good at while writing the language-specific logic in a higher-level language.

Please itself is entirely written in Go. We ran through a bunch of options for it originally and managed to eliminate all the others one way or another; fortunately the learning curve turned out not to be too steep and the performance as good as we’d hoped for. Initially we linked in some other dependencies (for example Python as the parser for BUILD files and some other tooling) but subsequently the core system has moved to pure Go (NOTE: there are some Java components used for compiling Java code, but they’re not needed for the core of Please).

Current state

Please is fully open source - you can check out the code on github, which is pretty easy to build from scratch (you mostly just need Go installed). If you don’t want to build it yourself, there’s a handy installation script that you can use by running:

curl https://get.please.build | bash

Right now it works fully on Linux (we’ve tested primarily on Ubuntu) and OSX, and FreeBSD has been known to work in the past with a bit of work.

There are instructions and documentation on the website giving more detail about how to use it.

In future I’ll be posting some more in-depth articles about specific aspects of the system, but for now this should give you a quick introduction to how it all works!