Event-Driven Video Processing on GCP: Killing the Request That Could Never Finish

Transcribing a screening video takes 2 to 5 minutes. A load balancer gives up on an HTTP request in about one. That arithmetic has no tuning parameter — any synchronous design for Navero's candidate video screening was dead before the first line of code, and the interesting work was accepting that and building around events instead.

The Naive Solution: One Big Request

The obvious design is a single endpoint that does everything: accept the upload, transcribe it, score it, return the result.

This fails in ways that no amount of tuning can rescue:

• Timeouts are structural. Transcription takes 2–5 minutes per recording; load balancers and browsers give up long before that. Raising limits just moves the cliff.
• Retries are ruinously expensive. The request is all-or-nothing. If AI scoring fails on the last step, the retry re-uploads and re-transcribes — you pay the vendor twice for one failure that had nothing to do with them.
• The slowest vendor sets your ceiling. Your API's latency, availability, and throughput are now whatever the transcription provider is having today.
• Scaling is hostage to waiting. Every in-flight screening pins a request for minutes. Server capacity is consumed by workers doing nothing but holding connections open.
• Deploys kill work in progress. Restart the API mid-request and the candidate's screening silently vanishes.

The Solution: An Event-Driven Pipeline

We inverted the design: the API's only synchronous job is to accept the upload. Everything downstream reacts to events.

1. Upload — Videos land in GCS (Google Cloud Storage); the endpoint returns 202 Accepted immediately
2. Trigger — GCS emits an event that invokes a Cloud Function
3. Transcribe — The function submits the video to AssemblyAI and gets out of the way
4. Webhook — AssemblyAI calls our webhook when transcription completes
5. Analyze — Another Cloud Function runs AI scoring on the transcript
6. Notify — The dashboard receives a real-time update via WebSocket

Why This Shape

Each arrow in that diagram is a boundary where work is durably parked before the next stage picks it up — and that is exactly what fixes the naive design's failures:

• Failure is scoped to a stage. If AI scoring fails, we retry scoring against the stored transcript. The video is not re-uploaded; the transcription is not re-purchased.
• Slowness stops propagating. A transcription backlog delays transcripts — it does not slow uploads, and it cannot take the API down.
• The client never waits on a vendor. 202 Accepted in under a second, then the client polls or receives a WebSocket push. Perceived responsiveness is decoupled from actual processing time.
• Deploys are safe. In-flight work lives in GCS objects, vendor queues, and events — not in the memory of a process being restarted.

What This Shape Costs

Event-driven is not free, and pretending otherwise is how teams end up resenting it:

• Debugging crosses five hops. A synchronous request has one stack trace; this pipeline has a GCS event, two Cloud Functions, a vendor webhook, and a WebSocket push. Without correlation IDs threaded through every stage, "where did screening X stall?" is archaeology.
• The dashboard is eventually consistent. Between upload and scored result there is a window where the truthful status is "somewhere in the pipeline." The UI has to be designed for that window instead of pretending it does not exist.
• Local development gets harder. You cannot run GCS event triggers and vendor webhooks on a laptop without emulators and tunnels; the dev loop is real work you are signing up for.

For a pipeline whose slowest stage takes minutes, these costs are easily worth it. For a flow that completes in two seconds, they are not — a queue and a worker, or even a plain synchronous call, would be the honest choice.

Key Lessons

Design for failure. Every step in the pipeline can fail independently. We added dead letter queues and retry policies at each stage — and because stages are idempotent, retries are safe.

Make it observable. An async pipeline hides failures that a synchronous request would have thrown in your face. We integrated error tracking across all Cloud Functions; when a transcription webhook fails at 2 AM, we know before the hiring manager notices.

Keep the API fast. The upload endpoint does one thing: durably accept the video. The moment work is safely parked, respond and let events carry it forward.

Results

• Video screening turnaround went from hours (manual) to minutes (automated)
• Zero data loss across 10,000+ processed screenings
• P99 upload latency under 500ms

Related Reading

• Scaling an Open-Source Chatbot to 700+ Stars — distributed coordination and horizontal scaling in another production system