Context
In July 2026 the old CI server was shut down, which left the learner platform with no release path. I took ownership of keeping production releasing.
The problem
Releases had to stay safe without CI and without credential key files on laptops. The interim path could not be trusted blindly: a bad release goes to every school.
What I did
I wrote local release scripts for the app, the loader and the live-class app. They authenticate with an attached service account, not key files.
After a version-collision incident on 28 July, a reused version plus a 6-month CDN TTL served stale bytes. I added a guard that checks the bucket listing as the source of truth.
After a stale-widgets incident on 6 October, where the CDN ignored no-cache, I invalidated and started a new version line.
I added a guard against the deploy script silently reverting submodule pins, which had broken Android DRM streaming.
I changed the loader release to open a PR instead of pushing to main, and added a check that the built artefact contains the injected i18n URL.
Key decision
I made the scripts assert against built artefacts and fail loudly, and I kept them as an interim until the cloud pipelines are trusted.
The trade-off is that releases still run from laptops with no CI, which is a standing risk.
Result
Production kept releasing; prod release #219 is one example. I do not have a count of missed releases to quote, and the laptop-release risk is still open.
What I learned
I would build the guards before the first incident. Both incidents were cheap to prevent with a check on stale versions.
Stack
- Bash
- Google Cloud Storage
- Cloud CDN
- Service accounts
- Git
