Skip to main content

ADR 0003: Use Firestore as the Job Queue

  • Status: Superseded in part
  • Date: 2025-10-15
  • Updated: 2026-07-23
  • Deciders: Akshay
  • Tags: data-model, infrastructure

Context​

Clients need real-time state for video jobs. The original choice was to use jobs/{jobId} both as the visible state machine and as the effective queue, avoiding a separate broker at low volume.

Original decision​

Firestore job documents were the queue and the client-visible state. Storage triggers advanced pending → running → done | failed, while clients subscribed to snapshots.

Current decision​

Firestore remains the authoritative state, ownership, retry, quota-release, and UI-subscription record. Dispatch is no longer Firestore-only:

  1. The Storage trigger validates the upload and advances the job.
  2. It enqueues the dispatchGpuJob Firebase Task Queue worker.
  3. The task has three attempts, 10–120 second backoff, and five concurrent dispatches.
  4. Terminal handlers use Firestore transactions so only one path wins.

This change supersedes the “no separate queue service” portion of the ADR, not the decision to keep durable client-visible job state in Firestore.

Consequences​

Positive​

  • Clients still receive real-time updates without polling.
  • Dispatch retries and concurrency limits no longer need ad hoc logic.
  • Job state remains easy to inspect and operate in Firestore/Admin.
  • claimJobDone and failJob make slot release idempotent.

Negative​

  • State and dispatch now span Firestore and Firebase Task Queue.
  • Operators must inspect both function/task logs and the job document.
  • A permanently failed dispatch still needs terminal failure/reconciliation.

Follow-up actions​

  • Add atomic terminal transitions.
  • Add hourly reconcileStuckJobs.
  • Add adminRetryJob.
  • Move dispatch to dispatchGpuJob Task Queue.
  • Re-evaluate throughput and dead-letter needs as volume grows.

References​

  • acesense-auth-function/video/gpu/enqueue.ts
  • acesense-auth-function/video/gpu/dispatch-worker.ts
  • acesense-auth-function/video/reconcile.ts
  • Job document