Context
This is the Android build runner for the ed-tech company's per-school app builds. I designed it and was the only implementer.
The problem
Builds ran on a managed container service. Each build spent about 6 minutes pulling its image, and the median run took about 24 minutes. The old path before that used long-lived machines.
What I did
I built a runner that starts one virtual machine per build from a prebaked disk image. The machine has no public address and holds no keys, and it deletes itself when the build ends.
It uses a compute-optimised machine type and fails over between zones if one is out of capacity.
When the fleet is at its cap, the trigger returns a 429 with a no-capacity reason, and the control plane queues the build.
I limited each machine to a 2 hour maximum run, and the result is read back from the logs.
I also locked the trigger to authenticated callers and had it log the exact source commit of each build.
Key decision
I picked per-build VMs over a pool of warm machines. A pool would start faster still, but it would bring back long-lived machines with credentials, which I was trying to remove.
The price is a cold start from the disk image on every build, and capacity that I manage with back-pressure.
Result
The median build fell from about 24 minutes to about 6 minutes, about 4 times faster than the managed service. The 6-minute figure is the median of 178 builds.
The quota allows roughly 124 VMs at once.
What I learned
I would again make the failure path part of the design. Capacity, zone failover and cleanup took as much thought as the speed.
Stack
- GCE
- Cloud Functions
- Node.js
- Cloud Logging
- Terraform
- IAM
