Context
Release Center is the deploy console behind 21 production pipelines on Google Cloud. I wrote most of its v2 and v3 interface and co-developed the API with the backend owner, a colleague.
The problem
Every release to dev previews, staging and production needed a self-serve console. Previews could not chain a producer app to a consumer app. Stale environments cost money. The orchestrator also had failure modes, including one that kept creating duplicate production builds.
What I did
I fixed an N+1 query in the running-deploys call. Cold load dropped from 79 s to 4 s.
I fixed the duplicate-build loop after a post-mortem: live runs are now checked against the build service, so an orphaned running record cannot block or repeat a build. The loop had caused 138 duplicate production builds.
I made one failed pipeline stop aborting the whole deployment; downstream steps are marked blocked instead.
I added a daily default environment and a 7-day preview reaper. One sweep closed 171 stale releases and freed 38 previews.
I reverse-specced the old tool into 142 requirements and 431 scenarios before porting.
Key decision
I wrote an as-built spec of the old tool before porting it to v3. It was worth the effort: it exposed 21 regressions.
The worst: for about a month, backend releases made in v3 ran no database migrations and no background-worker deploys. I fixed 17 and 2 partly.
Result
Cold load went from 79 s to 4 s. v2 and v3 both reached general availability within about 6 weeks. The duplicate-build loop ended.
What I learned
I would write the as-built spec before starting any rewrite. The missing-migrations bug would have shipped without it.
Stack
- React
- Google Cloud Deploy
- Cloud Build
- Cloud Workflows
- Node.js
