Ninja build with a Go microservices project

At work, we've been using the Ninja build tool to compile the dependencies of our Go microservices. Ninja builds your project, keeping track of what's changed so it doesn't repeat work.

Our dependencies

Our microservices depend on various bits of generated code, including:

• Protobuf and gRPC object definitions
• A GraphQL server
• Code for querying our SQL database

We don't commit this generated code because:

• Generated code (especially Protobuf definitions) was a frequent source of merge conflicts on a previous project. These are annoying because they're trivial to fix - just regenerate the files.
• It increases the amount of data stored in Git, which over time makes git commands slower.

Instead, we regenerate it when it might have changed. Ninja lets us do this efficiently.

Ninja file semantics

Ninja is configured with a file named build.ninja. I'll briefly cover its syntax, but you should also read the manual if you're interested in using Ninja yourself.

Ninja files contain two statement types:

1. Rules

These specify how to build a certain type of dependency. For example, a rule for building a go project might be:

rule gobuild
    command = go build -o $out ./$in

The variables $out and $in refer to the output and input files specified by particular build statements.

2. Build statements

These describe how to build a particular dependency. They map an output file(s) to the files used to create it, and the rule which creates them. E.g:

build mytool: gobuild mytool/main.go

Ninja uses build statements to work out what it needs to do. It only runs a build statement if the output is missing or one of the inputs has changed.

The build.ninja file

The ninja file has a minimal syntax, and it's expected that you use a higher level language to generate it.

We generate ours with a Go script, using text/template. We handwrite rules in the template file, and generate build statements programatically.

Our template looks something like:

type BuildStatement struct {
  Outputs      []string
  Rule         string
  Dependencies []string
}

func (b BuildStatement) String() string {
  return fmt.Sprintf(
    "build %s: %s %s", strings.Join(v.Outputs, " "), v.Rule,
    strings.Join(v.Dependencies),
  )
}

var tmpl = `
rule gobuild
    command = go build -o $out ./$in

{{- range .BuildStatements }}
{{ .String }}
{{- end }}`

Working out dependencies

A lot of our build tools work out their dependencies implicitly. For example, go build takes a directory containing a Go project to build, but Ninja expects a list of files to understand what's changed. We can express this with implicit dependencies. We run a glob in our Ninja file generator to find all these dependency files. Go's standard library Glob function doesn't support the ** syntax recursing into directories, so we use this zglob library.

Working out outputs

A lot of our build tools don't need the output file(s) to be specified explicitly. We thought about working these out dynamically (like we do for dependencies). For example, protobuf outputs files which all have the suffix pb.go or pb.<generator name>.go. However, this leads to a circular dependency between the build.ninja file and the project, where Ninja's understanding of what output files it expects depends on Ninja already having been run.

Instead, we manually write out the expected dependencies for each tool.

Running Ninja

We run Ninja after git pull and git checkout by creating two Git hooks (.git/hooks/post-merge, .git/hooks/post-checkout) which contain:

#!/bin/sh

# The script which generates build.ninja
generate-ninja-file

ninja

We don't commit the build.ninja file, but generate it before every use. This lets us customise elements build for the different contexts where it's run.