Skip to content
Keshav Raj
A database being craned to a new platform with an identical rollback copy and a rewind dial.
RELEASE YARDSBYTE-IDENTICAL
SHIPPED Production DB migration with byte-identical rollback in ~2 min.I OWNED Designer + operatorOUTCOME BYTE-IDENTICAL

Maintenance toolkit

Keshav RajDesigner + operator · 2026 · ExecutedContact meResume PDF

Context

In July 2026 I designed and operated the maintenance-mode toolkit for a production database migration. It flips a maintenance page on across the production load balancers for every school domain.

The problem

The migration needed a maintenance page in front of the whole product. A frontend rebuild takes about 1 hour, too slow to toggle in or out of a window. The change touches live global URL maps, and rollback had to be exact.

What I did

  1. I wrote numbered, idempotent scripts. They back up the current config with a fresh export and set up a maintenance backend bucket with the CDN off.

  2. I wrote a generator that rewrites only the 5 app matchers and prints the diffs.

  3. I loaded the generated map on a throwaway test load balancer first, and automated per-host checks that report OK, MAINT or NOT-MAINT.

  4. I added two maintenance-on variants, one keeping the API live and one flipping it too, plus a one-command rollback and a runbook.

Key decision

I toggled at the URL-map layer instead of in the app. It needs no rebuild, and the backup makes rollback a restore.

The cost is that unmatched paths return the maintenance body with a 404, by design. I accepted that since it is correct for a maintenance window.

Result

The window ran at 01:30 IST. The rollback was byte-identical to the original config. Propagation took about 2 minutes, and no CDN invalidation was needed.

What I learned

I would rehearse on a test load balancer again. I would keep the per-host verification for any change that touches live routing.

Stack

  • Google Cloud load balancing
  • Cloud CDN
  • Bash
  • gcloud
Play the world →