Skip to content
Keshav Raj
A release train still departing at night after a shut-down factory goes dark, with a keyless lock opening.
RELEASE YARDS0 MISSED
SHIPPED Kept production releasing with no CI and no key files.I OWNED OwnerOUTCOME 0 MISSED

Releasing after the CI shutdown

Keshav RajOwner · 2026 · InterimContact meResume PDF

Context

In July 2026 the old CI server was shut down, which left the learner platform with no release path. I took ownership of keeping production releasing.

The problem

Releases had to stay safe without CI and without credential key files on laptops. The interim path could not be trusted blindly: a bad release goes to every school.

What I did

  1. I wrote local release scripts for the app, the loader and the live-class app. They authenticate with an attached service account, not key files.

  2. After a version-collision incident on 28 July, a reused version plus a 6-month CDN TTL served stale bytes. I added a guard that checks the bucket listing as the source of truth.

  3. After a stale-widgets incident on 6 October, where the CDN ignored no-cache, I invalidated and started a new version line.

  4. I added a guard against the deploy script silently reverting submodule pins, which had broken Android DRM streaming.

  5. I changed the loader release to open a PR instead of pushing to main, and added a check that the built artefact contains the injected i18n URL.

Key decision

I made the scripts assert against built artefacts and fail loudly, and I kept them as an interim until the cloud pipelines are trusted.

The trade-off is that releases still run from laptops with no CI, which is a standing risk.

Result

Production kept releasing; prod release #219 is one example. I do not have a count of missed releases to quote, and the laptop-release risk is still open.

What I learned

I would build the guards before the first incident. Both incidents were cheap to prevent with a check on stale versions.

Stack

  • Bash
  • Google Cloud Storage
  • Cloud CDN
  • Service accounts
  • Git
Play the world →