Skip to content
Keshav Raj
Disposable build machines each assembling a phone then vanishing, with a speedometer and a queue gate.
PERFORMANCE DOCKS24 → 6 MIN
SHIPPED Keyless, self-deleting build VMs with back-pressure: 4× faster than managed builds.I OWNED Architect + sole implementerOUTCOME 24 → 6 MIN

Per-build VM runners

Keshav RajArchitect + sole implementer · 2026 · ShippedContact meResume PDF

Context

This is the Android build runner for the ed-tech company's per-school app builds. I designed it and was the only implementer.

The problem

Builds ran on a managed container service. Each build spent about 6 minutes pulling its image, and the median run took about 24 minutes. The old path before that used long-lived machines.

What I did

  1. I built a runner that starts one virtual machine per build from a prebaked disk image. The machine has no public address and holds no keys, and it deletes itself when the build ends.

  2. It uses a compute-optimised machine type and fails over between zones if one is out of capacity.

  3. When the fleet is at its cap, the trigger returns a 429 with a no-capacity reason, and the control plane queues the build.

  4. I limited each machine to a 2 hour maximum run, and the result is read back from the logs.

  5. I also locked the trigger to authenticated callers and had it log the exact source commit of each build.

Key decision

I picked per-build VMs over a pool of warm machines. A pool would start faster still, but it would bring back long-lived machines with credentials, which I was trying to remove.

The price is a cold start from the disk image on every build, and capacity that I manage with back-pressure.

Result

The median build fell from about 24 minutes to about 6 minutes, about 4 times faster than the managed service. The 6-minute figure is the median of 178 builds.

The quota allows roughly 124 VMs at once.

What I learned

I would again make the failure path part of the design. Capacity, zone failover and cleanup took as much thought as the speed.

Stack

  • GCE
  • Cloud Functions
  • Node.js
  • Cloud Logging
  • Terraform
  • IAM
Play the world →