Skip to main content

🚨 Runbook: RunPod API Errors

:::danger Not in use since 2026-08-16 RunPod is disabled. Lambda is the only GPU provider. Nothing dispatches to RunPod, so these symptoms should not occur — if you are seeing them, something has re-enabled the provider.

RunPod was a paid fallback that sat at 402 Insufficient Balance, which turned a dead Lambda worker into GPU_JOB_FAILED with the user's video intact in Storage rather than a slower run. The adapter is still in the tree; re-enabling means funding the account and naming runpod in GPU_PROVIDER_POLICY.enabled. Keep this runbook for that day. :::

:::warning Audit scope RunPod configuration and provider behavior belong to the excluded GPU backend and were not re-verified in the 2026-07-23 non-GPU audit. The older last_verified date is intentional. Confirm current provider configuration and API behavior before following commands on this page. :::

  • Severity: 🔴 Critical — no videos can be processed
  • Time to mitigate: 5-10 min (rollback) / 30-60 min (fix)

Symptom​

processVideoOnUpload logs show one or more of:

  • RunPod API error: 500 or 502
  • RunPod API error: 401 - Unauthorized
  • fetch failed — network error
  • Jobs get created but immediately fail, or stall with RunPod returning IN_QUEUE forever

Impact​

🔴 All new uploads fail to process. User-visible: spinner → timeout → "something went wrong". No mitigation without a working RunPod endpoint.

Diagnosis​

Step 1 — Verify the RunPod endpoint is alive​

  1. Open runpod.io/console/serverless
  2. Select the AceSense endpoint
  3. Check the health banner and recent requests

If the endpoint shows "No workers available" or "Endpoint disabled" → config issue (Step 3). If the endpoint shows recent failed requests → image issue (Step 4).

Step 2 — Verify secrets​

firebase functions:secrets:access RUNPOD_ENDPOINT_ID --project acesense-prod
firebase functions:secrets:access RUNPOD_API_KEY --project acesense-prod

Both must return values. If either is empty or stale (rotated on RunPod side but not updated in Firebase), the dispatch will fail with 401.

Step 3 — Check endpoint config​

  • GPU type: should be NVIDIA RTX 4090
  • Docker image: should match the latest known-good tag (e.g. akshaysarode/acesense:v46)
  • Max workers: 10
  • Min workers: 0
  • Idle timeout: 60s

Step 4 — Check a failing worker's logs​

On the RunPod Requests tab, click the latest failed job. Common errors:

  • ImportError / ModuleNotFoundError → image built with wrong Python deps
  • CUDA out of memory → video too large, or batch size too high
  • Timeout → job exceeded 10-min execution limit

Mitigation (make the pain stop)​

Option A — Rollback to last known good image​

  1. In the RunPod console, edit the endpoint
  2. Change the Docker image tag to the last known-good version (e.g. vN-1)
  3. Save — workers pick up on next cold start (new jobs within ~60s)

Option B — Rotate the API key if compromised​

# Generate a new key in the RunPod console first
firebase functions:secrets:set RUNPOD_API_KEY --project acesense-prod
# Paste new key when prompted

firebase deploy --only functions --project acesense-prod

Option C — Pause uploads​

If neither above helps, stop new uploads to prevent user frustration:

// Deploy a temporary Storage rule
match /videos/{path=**} { allow write: if false; }

Then revert once resolved. Post to #acesense-dev.

Resolution​

  • If rollback fixed it → diagnose why the new image broke (likely a PR merged to gpu-backend). Fix + rebuild + push new tag + update endpoint.
  • If secrets were wrong → update via firebase functions:secrets:set and redeploy.
  • If infra issue on RunPod's side → check status.runpod.io and wait it out. No user-side fix.

Prevention​

  • Run pytest tests/ before building a new Docker image
  • Tag Docker images with commit SHA, not just version, so rollback is unambiguous
  • Add a smoke test: after Docker push, fire a RunPod request with input_videos/sample.mp4 and verify the handler returns success
  • Wire up a RunPod health-check function that runs hourly and alerts on failure

🎯 Related​