Skip to main content

ADR 0001: Use RunPod Serverless for GPU Processing

:::note Audit scope (2026-07-23) This GPU-specific record was inventoried but not reverified because the GPU backend and provider configuration were excluded from the audit. Its last_verified date is intentionally unchanged. :::

  • Status: ✅ Accepted
  • Date: 2025-09-12
  • Deciders: Akshay
  • Tags: infrastructure, gpu, cost

Context​

We need GPU inference (YOLO, TrackNet, MediaPipe, CatBoost) for every uploaded video. Requirements:

  • Pay-per-use — early-stage volume is bursty; we can't afford always-on GPUs
  • NVIDIA CUDA — PyTorch + CatBoost need it
  • Docker-based deploy — easy to rebuild the image with new model weights
  • No vendor lock-in — the handler needs to stay portable
  • Low ops — one engineer maintains the whole backend

Options Considered​

A. GCP Vertex AI Custom Training / Prediction​

  • Pros: Same cloud as Firebase; IAM integrates cleanly
  • Cons: Long cold starts (5-10 min); opinionated about model format; more expensive for short jobs; quota approval friction
  • Cost: ~$0.92/hr for T4, ~$2.48/hr for V100 (V100 weaker than 4090)

B. AWS SageMaker Serverless Inference​

  • Pros: Mature, good autoscaling
  • Cons: CPU-only for serverless (GPUs require provisioned endpoints = always-on cost); cross-cloud adds complexity
  • Cost: Provisioned ml.g4dn.xlarge ~$0.74/hr always-on → $540/mo baseline

C. RunPod Serverless​

  • Pros: RTX 4090 available; true pay-per-second; fast cold starts (~30-60s); simple Docker-based deploy; REST API; cheap
  • Cons: Smaller company; less enterprise tooling; debugging is via console logs only
  • Cost: ~$0.00076/sec = $2.74/hr only when active

D. Self-hosted GPU box​

  • Pros: Fixed cost; full control
  • Cons: 24/7 electricity + hardware failure risk; we'd have to build our own job queue + worker pool; on-call burden
  • Cost: ~$2500 hardware + ~$30/mo power; zero marginal cost per job

Decision​

We will use RunPod Serverless for all GPU inference in production.

RunPod wins on cost-per-job at our current volume (< 5K videos/mo), gives us the strongest GPU available (RTX 4090 > V100/T4), and has the simplest Dockerfile-based deploy flow. The cold-start penalty (~30-60s) is acceptable since uploads are already async with real-time Firestore status updates.

Consequences​

Positive​

  • Near-zero idle cost — min workers stays at 0
  • RTX 4090 is faster than anything available on Vertex/SageMaker serverless
  • Deploy is docker push + console tag update — no cloud-specific APIs
  • Portable handler (runpod_handler.py) — could swap to another provider in ~1 day

Negative​

  • RunPod outages affect all processing with no fallback
  • No GCP-native IAM — we pass credentials via environment variables (stored as Firebase secrets)
  • Debugging production jobs requires the RunPod web console — no log streaming to Firebase
  • Cold starts hurt first-after-idle UX; we mask this with progress indicators

Neutral​

  • Requires a separate billing relationship with RunPod
  • We need a Docker Hub account for image hosting (akshaysarode/acesense)

Follow-up Actions​

  • Build runpod_handler.py
  • Deploy endpoint to production
  • Set RUNPOD_ENDPOINT_ID + RUNPOD_API_KEY secrets
  • Add a fallback path: if RunPod returns 5xx three times, surface a user-friendly error (currently fails silently)
  • Set up RunPod billing alerts at $100, $500, $1000/mo

References​