Audit — 2026-08-03

Frozen snapshot — do not update. Kept for how a decision was reached, not for what is true now. Current state lives in Architecture → Current State.

Live-production audit: every check below was executed against prod, not inferred from code. Companion to ANALYSIS-2026-07-28.md (architecture) and AUDIT-2026-07-25.md.

Verdict

Core product functions. One provider outage (RunPod credits), two unexplained pipeline crashes, and a handful of latent issues. Nothing user-facing is down right now, but fallback capacity is gone — availability rides on the Lambda scaler alone until RunPod is topped up.

What was checked and passed

Check	Result
Cloud Run services	54/54 healthy (3 Stripe fns carried failed revisions from the broken-deps era — clean-redeployed during this audit)
Landing / admin / api / docs hosts	all 200, < 0.6 s
/link Android	302 → public Play listing (io.acesense.app) ✓
/link iOS	302 → App Store ✓
stripeWebhook bad signature	400 (correct rejection) ✓
Last-7d jobs	40 total: 30 done, 10 failed (6 client-upload, 4 GPU-side — see findings)
videoUrl on done jobs	30/30 present; newest streams 206 video/mp4 with Range ✓
Notifications	analysis_ready created for newest job (08-03 00:58) ✓
Region filesystems	4/4 md5-identical, creds intact — re-synced twice since, final at 8e0cc68
Web-preview fix	verified end-to-end on a GPU box: 2160×3840 source → 406×720 render
Container-deps guard	verify:container in predeploy; proven to catch the @firebase/app outage
Firestore hygiene	6 dead workers/ docs deleted; scaler config note refreshed

Remediation pass (same day, after the audit above)

Finding	Outcome
F2 — 2 pipeline crashes	ROOT-CAUSED AND FIXED. Both were the audio-tail bound, set at 0.25 s, missed by 0.250068 s and 0.269735 s (~12–13 AAC frames). The original bound covered only the trailing partial frame, not encoder delay + priming. Raised to 0.5 s (989ec0f), deployed to 4/4 regions, and the first video now completes where it crashed twice.
F3 — LBJr7C7F casualty	RECOVERED — requeued, done.
F4 — tokenless resultUrl	FIXED + BACKFILLED. Finalize now mints a token (onResultUploaded deployed); 84/84 done jobs repaired (an earlier pass fixed 65; a second swept the remaining 12 whose result objects needed a token minted), verified streaming 206.
Leaked GCP worker SA key	ROTATED. New key 82a6ccea minted, distributed to 4/4 region filesystems, worker heartbeat confirmed on it, then the leaked key 1b7888ed DISABLED in IAM.
Lambda key v1 still bound	RESOLVED — lambdaScalerTick + dispatchGpuJob redeployed; every live revision now binds v2.
5 repos holding 07-25 hardening	PUSHED (secret-scanned first).
Landing serving a Jul 23 build	REDEPLOYED, live label verified.
pnpm-9 workspace break	FIXED in admin/annotate/launchpad; 280 previously-unrunnable tests now green.

RunPod status (updated 2026-08-03):

1. FIREBASE_SERVICE_ACCOUNT_BASE64 — DONE. Akshay pasted the rotated key (82a6ccea) into the RunPod dashboard; the staged local copy at ~/Developer/secrets/runpod-firebase-sa-base64.txt has been deleted. RunPod exposes secrets read-only over its API (myself { secrets { name } } works; 11 mutation names and 3 REST paths all rejected), so the value itself cannot be read back to verify — only its presence, which is confirmed. First RunPod job after credits return is the real test; if it fails auth, re-stage from /lambda/nfs/acesense-worker/creds/.acesense-worker-sa.json on any region filesystem.

2. RunPod credits (F1) — still outstanding. Endpoint 0jcdupawiv1e1m answers 402 Insufficient Balance, so the pool has no fallback: any job arriving with no live Lambda worker fails as GPU_JOB_FAILED. Deferred by Akshay.

Capacity change — Lambda fleet replaces the RunPod fallback (2026-08-03)

F1 exposed a design assumption, not just an empty wallet: the pool was built so that "no idle worker right now" meant "send it to RunPod". Two limits kept Lambda at effectively one worker —

• the scaler launched at most ONE instance, and only when there were ZERO idle workers and nothing booting;
• the pool refused past 2 active queue docs and threw, routing overflow to RunPod.

Both are gone (6a7d90e, deployed). launchesNeeded counts idle workers plus booting instances as capacity and launches one box per uncovered job, bounded by maxAutoInstances and per-tick maxLaunchesPerTick; each launch walks the rung ladder independently so one region's capacity error does not sink the rest. The pool now queues to maxQueueDepth before refusing.

Live config: maxAutoInstances 1 → 6, maxLaunchesPerTick 2, maxQueueDepth 12, 8 rungs across 4 regions. RunPod stays wired and takes traffic whenever healthy — it is no longer the answer to a busy minute.

Observability — every run now accounts for itself (2026-08-03)

Tracing one real 230 s upload showed the analyzer reporting total_time 113 s against a 279 s job: a 166 s gap nothing recorded, because CFR normalization, orientation, portrait adaptation, the analysis resize and finalization (which renders the whole web video and every shot clip) were untimed. All five are now timed into the published processing block alongside a wall_total that reconciles to the worker's observed duration (8e0cc68). Verified live on a GPU box: wall 24.61 s = 3.96 cfr + 0.41 orientation + 0.01 portrait + 0.01 resize + 16.20 analysis + 3.78 finalization.

Failures used to write {"reason": "processing_error"} and nothing else, which is why diagnosing two of them cost a GPU box and a manual re-run each. The phase timer records in finally, so the rejection file now carries failedPhase, timings, errorType, errorMessage and wallSeconds; handleGpuRejection lifts them onto the job's failure field (b0a3680).

Per-stage timings also land on the job doc (timings, finalizationSteps) from the parse finalize already does — so fleet-wide questions like "is shot detection getting slower" are one Firestore query instead of downloading every analysis JSON.

Findings, ranked

F1 — RunPod fallback is dead: 402 Insufficient Balance (SEV: high)

https://api.runpod.ai/v2/<endpoint>/health returns insufficient-balance. Every job that arrives while no Lambda worker is alive follows lambda-unavailable → RunPod → 402 → retries → GPU_JOB_FAILED. Two users hit exactly this (k326RvIv… 07-29, recovered; LBJr7C7F… 08-02, still failed). Action (Akshay only): top up RunPod credits. Until then the pool has no fallback; the scaler is the only capacity source.

F2 — Two pipeline crashes on current code, cause unknown (SEV: high)

Both files are complete in GCS (byte counts match declared), both crashed the pipeline (processing_error):

• videos/nqB1zoAdqwU0C8RInshpRUcDjB23/cd5595b4380dbca8/… — 14.7 s, 0.75 MB
• videos/hq4WlijyIoZ6ej8ScUEJ63sfRIX2/6cf9a2901213fadc/… — 19.9 min, 118 MB

Same recovery playbook as 07-28 applies (requeue through a worker, read the journal). Needs a GPU box to reproduce; not run during this audit.

F3 — LBJr7C7F… (08-02) still failed with its video intact (SEV: med)

Casualty of F1. Recoverable via the requeue playbook once a worker is up — video is complete in storage.

F4 — resultUrl on job docs is tokenless → 403 (SEV: low, latent)

The GPU uploader sets no download token on _combined.json, and finalize falls back to a tokenless URL. The app is unaffected (it downloads via the authenticated Storage SDK using resultPath), but any consumer trusting resultUrl gets 403. Fix opportunistically: mint a token in finalize the way the web-video path does.

F5 — Standing items unchanged

• 5 leaked keys in gpu-backend pushed history — rotation runbook exists, still the top security debt. Lambda key v1 is still bound to scaler/dispatch until their redeploy (v2 exists; do not revoke v1 yet).
• Billing placeholders RESOLVED. RevenueCat was removed entirely (unused — zero RevenueCat records across 317 users; the 4 premium tiers are manual proGrantReason grants) and revenuecatWebhook deleted from europe-west1. It was the last binder of REVENUECAT_WEBHOOK_SECRET, so no function binds a placeholder now; STRIPE_API_KEY is at an enabled v2. A full --only functions deploy is no longer gated.
• Stripe wave is test-mode-gated by design; live billing is a code change.
• 12 of 16 Lambda region filesystems unseeded.
• Client upload reliability: 6 of 10 failures this week are uploads that never completed — app-side, unchanged.

Fixed during this audit

• 3 Stripe functions clean-redeployed (54/54 now green).
• 6 dead workers/ docs removed.
• config/gpuScaler note corrected (was still claiming se1 stale at 9559bdf).
• Landing hosting redeployed — acesense.io was serving a Jul 23 build; the "Android beta" labels fixed on 08-02 had never gone live.
• pnpm-workspace.yaml made pnpm-9-loadable in admin/annotate/launchpad — until then pnpm test could not run at all in those repos on this machine.

---

Per-repo deep audit

Repo	Tests	Build	Deploy parity	Notes
gpu-backend	1046 ✓ (+ruff)	Dockerfile paths valid	4/4 region FS at 8e0cc68 ✓	RunPod image untestable while credits at 0 (F1)
auth-function	571 ✓ (+lint, +verify:container)	tsc ✓	54/54 fns green; secrets/params complete	F4 fixed + backfilled
frontend	263 ✓, flutter analyze clean	—	n/a (store builds)	9 dirty files + 1 unpushed = deliberate 1.0.18 WIP (version bump, faststart sample, Kotlin migrator flags) — left untouched
landing	118 ✓	prerender ✓	was stale since Jul 23 → redeployed now	live label verified post-deploy
admin	80 ✓	vite ✓	live bundle Bt5J1aJd ≠ HEAD build Ci1BN-Mw — the 07-25 "in-progress console work + report-only CSP" commit is not deployed	flagged, not deployed: commit self-describes as in-progress
annotate	179 ✓	vite ✓	local-only tool (no hosting config)	—
launchpad	21 ✓	vite ✓	serves at acesense.io/launchpad → 200 ✓	—
docs	n/a	—	acesense.io/docs → 200 ✓	—
api-hosting	n/a	—	api.acesense.io 200; /v1 rewrites to apiserver; HSTS/nosniff/DENY headers present	root health at direct fn URL 200
brand	n/a	—	n/a	secret scan clean

Git hygiene: RESOLVED. All five were secret-scanned and pushed during the remediation pass. 9 of 10 repos are now clean and in sync with origin; the exception is acesense-frontend, which holds a deliberate 1.0.18 work in progress (version bump, faststart sample asset, Kotlin migrator flags) and was left untouched on purpose — committing someone else's in-flight release prep is not cleanup.

Firestore bookkeeping was also swept: workers/ and scalerInstances are both empty, matching a fleet of zero live instances, and no job sits in a non-terminal state.