Skip to main content

🚨 Runbook: Job Stuck in processing

  • Severity: 🟡 High
  • Time to mitigate: 5-10 min
  • Time to root cause: 30-60 min

:::tip Automatic recovery now exists Since the June 2026 reliability work, the hourly reconcileStuckJobs scheduled function fails jobs stuck in pending/uploaded (> 2 h) or processing (> 6 h) authoritatively — releasing the owner's concurrency slot and refunding API-job cost. This runbook is for when you can't wait for the sweep, when the sweep itself is failing, or when many jobs are stuck at once. :::

Symptom​

A user uploaded a video, the jobs/{jobId} document was created, it transitioned to status: "processing", but never reaches done. The Flutter app shows "Still processing…" indefinitely (the client deliberately never times out processing — the server sweep owns failure).

Impact​

  • ❌ Affected user can't see their analysis
  • ❌ GPU credit appears consumed but no result
  • ✅ Other users unaffected (usually)

Escalate to 🔴 Critical if more than one user is affected simultaneously — this suggests a Lambda fleet outage or a deploy bug.

:::note Provider Jobs run on the Lambda pull-worker fleet. RunPod was removed on 2026-08-16, so log lines mentioning it are historical. A dead worker is requeued onto the Lambda queue, and retryInfraFailedJobs re-dispatches infrastructure failures automatically every 10 minutes. :::

Diagnosis​

Step 1 — Find the jobId​

From the user, or from the Admin panel → Jobs tab → filter by userId + most recent.

Step 2 — Check the Firestore doc​

# In Firebase Console → Firestore → jobs → {jobId}
# Or via CLI (requires firebase-tools)
firebase firestore:documents:get jobs/{jobId} --project acesense-prod

Note:

  • status (should be running)
  • videoPath (e.g. videos/<sessionId>/<filename>.mp4)
  • resultPath (e.g. results/<sessionId>/<sessionId>_combined.json)
  • updatedAt — how long has it been stuck?

Step 3 — Did the video actually upload?​

gsutil ls gs://acesense-prod.firebasestorage.app/{videoPath}

If missing → the upload never completed. This is a client-side issue, not a pipeline issue. Jump to resolution option A.

If present → continue.

Step 4 — Did processVideoOnUpload fire?​

firebase functions:log --only processVideoOnUpload --project acesense-prod | grep {sessionId}

Look for:

  • 📹 Video uploaded: videos/{sessionId}/... — trigger fired ✅
  • 🚀 Sending to RunPod endpoint: ... — dispatched to RunPod ✅
  • ✅ RunPod job created: <runpod_job_id> — RunPod accepted the job ✅

If the trigger never fired → Storage trigger is broken. Jump to resolution option B.

If RunPod error logged → Jump to RunPod errors runbook.

Step 5 — Did the result file get written?​

gsutil ls gs://acesense-prod.firebasestorage.app/{resultPath}

If missing → RunPod is still working on it or it crashed. Check RunPod console:

  1. runpod.io/console/serverless
  2. Select the AceSense endpoint
  3. Requests tab → find the job by timestamp
  4. Check its status + logs

If present → onResultUploaded trigger didn't fire. Jump to resolution option C.

Mitigation (make the pain stop)​

Immediate (unblock the user)​

  1. Check if the result file exists (Step 5). If yes → manually mark the job done:

    Admin Panel → Jobs → find job → click status badge → "Mark Complete"

    Or via a one-off console script (requires admin cred):

    await admin.firestore().collection('jobs').doc('{jobId}').update({
      status: 'done',
      progress: 100,
      resultUrl: '<tokenized storage URL>',
      completedAt: admin.firestore.FieldValue.serverTimestamp(),
      updatedAt: admin.firestore.FieldValue.serverTimestamp(),
    });

  2. If the result file doesn't exist and RunPod shows the worker crashed:

    • Mark the job status: "failed" with a failureReason
    • Tell the user to re-upload (there's no automatic retry today)

If many users affected​

  1. Rollback recent deploys — firebase functions:log to identify the first failed job, check deploy history

  2. Disable uploads temporarily by deploying a rule that rejects new files to videos/:

    match /videos/{path=**} { allow write: if false; }

    Revert once root cause is identified.

  3. Post to Slack #acesense-dev: "Pausing video uploads — investigating".

Resolution (fix the root cause)​

A. Upload never completed​

  • Check the user's client logs if available (Sentry / Crashlytics not wired yet — ask the user to send a screenshot)
  • Common causes: network interruption, FFmpeg crash during chunking, insufficient device storage
  • Fix: tell user to retry; long-term, add better client-side error reporting

B. Storage trigger not firing​

  • firebase functions:log --only processVideoOnUpload shows no 📹 Video uploaded lines for this file
  • Check if the function is deployed: firebase functions:list --project acesense-prod
  • Check deploy status: Firebase Console → Functions → find processVideoOnUpload → status should be "Active"
  • Check path filter — filePath.startsWith("videos/") — is the path correct?
  • Fix: redeploy the function: cd acesense-auth-function && firebase deploy --only functions:processVideoOnUpload

C. onResultUploaded didn't fire​

  • Result file exists in Storage but no Firestore update
  • Common cause: file path doesn't match results/ prefix or doesn't end in _combined.json
  • Check the file path: is it results/<sessionId>/<sessionId>_combined.json exactly?
  • Check the query: jobs.where('resultPath', '==', <path>) — does the value in the job doc match the actual file path?
  • Fix: redeploy the function, or if a specific job is mis-pathed, update the resultPath field manually and re-upload the file to trigger the function again

Prevention​

  • Add a scheduled function that scans jobs for status == "running" with updatedAt > 30 min ago and marks them failed with a user-friendly message
  • Add a "retry" button to the Admin panel that re-enqueues a failed job without re-uploading
  • Wire up Sentry / Crashlytics in the Flutter app so upload failures surface automatically
  • Consider a periodic heartbeat write from RunPod to the job doc (progress updates) so stuck jobs are obvious

Postmortem​

If this incident affected more than one user OR took more than 30 minutes to mitigate, write a postmortem. See the postmortem template (once it exists).

🎯 Related​