Skip to main content

Postmortem: RunPod Image Regression (v47)

:::warning Audit scope This is a historical record whose primary evidence belongs to the excluded GPU backend. It was not re-verified during the 2026-07-23 non-GPU audit, so its older last_verified date is intentional. Preserve the original timeline; label any later correction explicitly. :::

FieldValue
Date2026-03-14
Severity🔴 Critical
Duration2h 18m (09:42 → 12:00 UTC)
Affected users100% of new uploads during the incident window (~18 users)
Triggered byGPU backend deploy (v47 Docker image)
Discovered byUser complaint in support channel
Authors@akshay
Status✅ Published

TL;DR​

A new GPU backend image (akshaysarode/acesense:v47) was pushed to RunPod at 09:42 UTC. The image had a silent regression in a model-loading path that only triggered on videos shorter than 60 seconds — our local smoke test used a 30-second clip, but ran with the dev cache enabled, which masked the bug. All short videos uploaded to production failed for 2h 18m until a user reported it and we rolled back to v46.

Timeline (UTC)​

TimeEvent
09:42docker push akshaysarode/acesense:v47 — new image live
09:44RunPod endpoint updated to v47
09:47First production video fails — silent, no alert fires
10:30More failed uploads — still no alert
11:05🚨 User @beta-user posts in #acesense-dev: "my upload has been stuck for an hour"
11:07@akshay opens RunPod console — sees 14 failed jobs with identical stack trace
11:14Diagnosis: error is in src/players/__init__.py — new FasterRCNN init path crashes on short videos
11:20Decision: rollback instead of forward-fix (couldn't root-cause in < 30 min)
11:25RunPod endpoint tag reverted v47 → v46
11:28New uploads start processing successfully
11:40Retried the 14 failed jobs manually via admin panel
12:00✅ All 14 affected uploads now have results
14:30Root cause PR opened (local test with --dev cache was hitting a stale code path)
Next dayv48 with fix + a proper smoke test shipped

Impact​

  • User-visible: Every upload between 09:47 and 11:25 got stuck in running. Users saw an indefinite spinner. No error surfaced.
  • Data loss: None. All 14 videos were re-processed after rollback; their analyses were identical to what they would have been.
  • Cost impact: ~$2 of wasted GPU compute on failed jobs. Negligible.
  • Trust impact: One user sent a frustrated message in Slack. No churn, but the session was painful.

Root Cause​

Proximate cause: src/players/__init__.py line 42 raised AttributeError when the ball trajectory was shorter than 60 frames. The new code assumed a minimum 2-second temporal window after a refactor, but never validated.

Contributing factors:

  1. Local smoke test ran with --dev — the dev cache short-circuits player detection entirely, so the bug was masked
  2. No CI for the GPU backend — pytest exists but isn't run on PR
  3. No production smoke test — after pushing an image, we didn't fire a test inference against it
  4. No alert on function error rate — processVideoOnUpload was logging RunPod API error lines, but nothing was watching
  5. Min workers = 0 — all new jobs hit the new image immediately; there was no canary

Why we didn't catch it earlier:

  • The failure was silent to users (just a longer spinner)
  • Our only detection was a user complaint in Slack — no alerting

Mitigation​

  1. docker pull akshaysarode/acesense:v46 to confirm the old image was still available
  2. In RunPod console → Endpoint → edit → change tag v47 → v46 → save
  3. Wait ~60s for new workers to spin up with the old image
  4. Test with a known-good video upload → ✅ working
  5. Open Admin panel → Jobs tab → filter status == failed && createdAt > 09:47 → click "Retry" on each (14 jobs)
  6. Post status update in #acesense-dev

What went well​

  • Rollback was a single click once the bad image was identified
  • Admin panel's manual retry worked on every failed job
  • No data loss — the raw videos were still in Storage
  • @akshay happened to be online and saw the Slack message within 2 minutes

What went poorly​

  • 2 hours passed before anyone noticed
  • The local test run passed — false confidence
  • No automation for the rollback steps — all clicks
  • Users had no indication their upload failed (the Flutter UI still showed progress)

Where we got lucky​

  • The bad image only affected new jobs — anything already running was on the old image
  • A vocal beta user alerted us — paying users might have churned silently

Action items​

#ActionOwnerDuePriorityStatus
1Add Cloud Monitoring alert on processVideoOnUpload error rate > 5% / 5 min@akshay2026-03-21🔴 P0✅ Done
2Add a deploy smoke test: after Docker push, run an input_videos/sample.mp4 inference against the new tag and verify exit 0@akshay2026-03-28🔴 P0✅ Done
3Run pytest tests/ on every GPU backend PR via GitHub Actions@akshay2026-04-05🟡 P1🟡 In progress
4Set RunPod min workers = 1 during business hours as a canary buffer@akshayTBD🟢 P2⚠️ Deferred (cost tradeoff)
5Surface upload failures to the Flutter UI (currently silent)@akshay2026-04-12🟡 P1🟡 In progress
6Add this incident to the Job Stuck runbook as a known cause@akshay2026-03-15🟢 P2✅ Done

Lessons Learned​

  • Local tests must match production. Running with --dev is for iteration speed, not validation. The smoke test that gates a push should run the exact same code path as production.
  • If it's not monitored, it's not monitored. We had all the log data but no one was reading it. An alert with a pager beats a dashboard no one looks at.
  • Users should never be the first to notice. Every minute between "it broke" and "we noticed" is trust bleeding out.
  • Rollback is not a plan B. It's plan A for most incidents — make it fast and scriptable.

Supporting data​

  • Failed job IDs: fd4a1, e9b3c, … (14 total — retrieved from Firestore jobs with query status == failed && createdAt > 09:47)

  • Stack trace (excerpt):

    File "/app/src/players/__init__.py", line 42, in load_model
      trajectory_window = features[:60]
    AttributeError: 'NoneType' object has no attribute '__getitem__'

🎯 Related​