Context
An ed-tech company runs an analytics ingest service that takes events from web, mobile and server and webhooks. It handles about 6 to 9.5 million requests a day. I was the primary engineer on its hardening in 2026.
The problem
The service was hard to observe. When events failed, nobody could say which clients or routes were failing or why.
Some background jobs, such as attendance report processing, also lost work when they crashed.
What I did
I built an ops dashboard with daily KPIs, a browser for failed events, and a request-error breakdown by status, route, reason and device.
I added a dead-letter store so failed events are kept instead of dropped, and made attendance processing recover from crashes, with bounded flood control.
I raised production minimum replicas from 1 to 2, so one pod loss no longer stops ingest.
I gated builds on tests and deployed by image digest, and retired the legacy tag-based pipeline.
Key decision
I kept the legacy wire format exactly as it was, and built a compatibility gate of 1,944 cases that compares the new validator against the old one. Downstream data partitions depend on that format, so any drift would be a silent data change.
The cost was carrying old behaviour forward on purpose, instead of tidying it.
Result
The compatibility gate shows 0 diffs across 1,944 cases. The service handles 6 to 9.5 million requests a day.
The dashboard made it possible to see that about 200,000 requests a day were failing with 401 because of client token bugs.
What I learned
I would build the observability first again, before touching the code. Without the breakdown, the 401 problem would have looked like noise.
Stack
- Node.js
- Fastify
- Redis
- Firestore
- Pub/Sub
- GKE
- Cloud Build
