Context
In July 2026 I designed and operated the maintenance-mode toolkit for a production database migration. It flips a maintenance page on across the production load balancers for every school domain.
The problem
The migration needed a maintenance page in front of the whole product. A frontend rebuild takes about 1 hour, too slow to toggle in or out of a window. The change touches live global URL maps, and rollback had to be exact.
What I did
I wrote numbered, idempotent scripts. They back up the current config with a fresh export and set up a maintenance backend bucket with the CDN off.
I wrote a generator that rewrites only the 5 app matchers and prints the diffs.
I loaded the generated map on a throwaway test load balancer first, and automated per-host checks that report OK, MAINT or NOT-MAINT.
I added two maintenance-on variants, one keeping the API live and one flipping it too, plus a one-command rollback and a runbook.
Key decision
I toggled at the URL-map layer instead of in the app. It needs no rebuild, and the backup makes rollback a restore.
The cost is that unmatched paths return the maintenance body with a 404, by design. I accepted that since it is correct for a maintenance window.
Result
The window ran at 01:30 IST. The rollback was byte-identical to the original config. Propagation took about 2 minutes, and no CDN invalidation was needed.
What I learned
I would rehearse on a test load balancer again. I would keep the per-host verification for any change that touches live routing.
Stack
- Google Cloud load balancing
- Cloud CDN
- Bash
- gcloud
