Analysis — 2026-07-28

Frozen snapshot — do not update. Kept for how a decision was reached, not for what is true now. Current state lives in Architecture → Current State.

Date: 2026-07-28, updated 2026-07-29 · Scope: every component under AcesenseProd/, production Firestore/GCS state, Lambda fleet, and the evidence from this week's 64-video corpus run.

2026-07-29 addendum — five-day failure audit and recovery. All 11 failed jobs from Jul 23–28 were diagnosed: 8 were abandoned client uploads with no bytes in storage (nothing to analyze; correctly failed), 3 were GPU-side and recoverable. Recovering them surfaced a fourth normalization defect — a GoPro clip whose video track edit-list holds the last frame until the audio ends failed output validation as a container mismatch (31.031s != 30.497s) after stream copy. Fixed (a82c7f0), mutation-tested, verified on the real file, deployed to all four seeded regions (us-southeast-1 included — its drift is closed), and pushed. The 31-second job re-ran in 90 s and the user received the production analysis_ready notification ("Your match report is ready"). The two 101-minute re-runs are the encode-ceiling fix's first production exercise.

Final outcome (07-29): all three recoverable jobs are done at progress 100 with live result URLs, and all three analysis_ready notifications were created through the production finalize path — including both 101-minute AV1 sessions (54 shots, match @ 0.675). The recovery also exposed and fixed a fifth defect: RUNNING_MAX_MS = 3 h in the lambda reconciler requeued a healthy 5-hour run three times and failed the job while the worker finished and uploaded the complete analysis. Fixed to 6 h (aligned with the stuck-job sweeper), 546 function tests green, committed 358f873; functions deploy deferred at Akshay's request pending the three missing billing secrets (STRIPE_API_KEY, STRIPE_WEBHOOK_SECRET, REVENUECAT_API_KEY — values live in their dashboards). Until deployed, the 3 h transport cap remains live in prod and any upload needing >3 h of wall time will fail despite the pipeline fixes. NVDEC hardware decode was implemented, measured at 0 to −3% (the pass is bound by the lossless ultrafast x264 encode; the fleet has no NVENC), and reverted per the equivalence-gate convention — numbers recorded in the gpu-backend CHANGELOG.

Two operational truths this recovery bought: worker processes cache imported modules across jobs (a filesystem code drop needs a worker restart before the next claim), and the hourly stuck-job sweeper reads updatedAt?.toMillis — tooling that resets job docs with ISO strings sends it to the original createdAt and it fails healthy re-runs. Also: multi-GPU SXM shapes fail bootstrap's ONNX-CUDA gate with CUDA error 802 without fabric-manager — if the scaler ladder ever lands on gpu_8x_v100 or 2x H100, workers will boot-timeout. Single-GPU shapes unaffected.

---

1. System map

Flutter app (acesense-frontend)
   └─ upload chunks → GCS videos/…  +  jobs/{id} in Firestore
Firebase Functions (acesense-auth-function, europe-west1)
   ├─ preflight (Gemini, fail-open)
   ├─ dispatch router: lambda FLEET (primary, autoscaled) → RunPod fallback only
   ├─ lambda-scaler (launch/kill boxes, ladder config), lambda-reconcile (re-dispatch stale)
   ├─ billing (Stripe + RevenueCat), quotas, abuse limits, moderation, retention/purge
   ├─ coach mode (links/consent/seats), insights, shot-classify (Gemini), MCP server
GPU workers (acesense-gpu-backend, PyTorch)
   ├─ Lambda pull-workers: region filesystem = code+venv+creds, bootstrap.sh, systemd
   └─ RunPod: image built on git push to main
Results → GCS results/… → onResultUploaded finalizes job → app renders analysis
Secondary: admin console, landing, docs, annotate, api-hosting, launchpad, brand

Capacity note (2026-08-03): the Lambda fleet is now the primary path — the scaler launches one box per uncovered queued job up to maxAutoInstances (6) instead of overflowing to RunPod, which is a paid fallback that can and did run out of credit.

Two deploy paths, only one follows git. RunPod tracks main automatically. The Lambda pool deploys by hand-copying the tree onto each region filesystem — no git, no CI. This split caused a 92-hour worker outage (stale region + broken bootstrap) and let a fixed bug keep firing from an unsynced region. Now documented in the gpu-backend CHANGELOG, but the mechanism itself is unchanged.

2. Current health (production numbers)

Metric	Value
Last 100 jobs	54 done / 16 failed (70 total in window)
Last 7 days	35 jobs, 24 done = 69% success
Failure taxonomy	13 × STUCK_JOB_TIMEOUT (client upload never completed) · 2 × GPU_REJECTED (encode-ceiling bug, fixed today) · 1 × GPU_JOB_FAILED (creds outage, fixed 07-27)
Lambda regions seeded	4 of 16 (us-east-1, us-west-1, us-south-2, us-southeast-1)
Region code state	e1/w1/s2 = 6182434 (current). se1 = one commit behind (no capacity to mount; still has the 3600 s encode ceiling)
GPU backend tests	1037 passed, 2 skipped; ruff clean
Auth function tests	34 test suites (billing, dispatch, reconcile, SSRF, quotas, coach, …)

Read the taxonomy carefully: 13 of 16 recent failures are the app failing to finish uploads, not the backend. The backend's own failure modes this week were both found, fixed, verified on the real failing files, and deployed (except se1).

3. What the corpus run established (64 real user videos, 42 users)

1. The pipeline runs in two modes that disagree about identical evidence. When court geometry resolves, stroke verification is skipped: 14 court-present videos produced 89% of all 1215 published shots (76.9/video), including 211 typed unknown — the same classifier abstention that is a rejection reason on the gated path. The other 44 videos kept 143 of 2505 candidates (5.7%).
2. Court is usually seen, often not solved. The 11 partial videos observe the court in 79–100% of frames yet solve 0–1 frames of homography, then get the strictest gate. Two 13-minute sessions with coverage 0.989/0.990 published zero shots and told the user to re-film. The defect is the homography solver, not detector recall.
3. Users retry when they get nothing. 9 of 64 uploads are byte-identical re-uploads of 4 files, every one following a disappointing result (zero shots, or a failure). Re-upload rate is a free product-health metric.
4. 24 of 64 videos published nothing. 14 legitimately (no court/rally visible), but the 4 "part of the court" cases are good footage lost to #2.
5. Every over-29.5-minute video was running on borrowed time: the CFR encode ceiling granted less than its own formula required; survival depended on resolution (85-min 640×360 passed; 101-min 1080p died twice). Fixed; ceiling now derives from the 7200 s server cap.

4. Defect ledger — this week

Defect	Status	Verification
CFR audio-tail tolerance killed 22-min upload over 113 ms	fixed 20d0b6b	real file → 286 shots
Playback frame count 792 != 791 killed 4K HEVC	fixed 41c6150	real file finalizes
Encode ceiling 3600 s killed 101-min 1080p (twice, same user)	fixed 6182434	mutation-tested both ways; live budget check per region
bootstrap.sh: creds wiped by rsync, py3.10 ordering	fixed 7a091ac	full bootstrap on fresh 24.04 box
Test pinned the clamped value at exactly the ceiling's breakeven (1800 s)	fixed in 6182434	now asserts the rate at 6079 s
GPU OOM on one corpus video	not a defect — my own GPU-packing during the sweep; solo run: 380 shots
Truncated upload rejection (587/1101 frames)	correct behavior

5. Security posture

• Open (P0): 5 leaked keys in gpu-backend pushed history — prod GCP SA key, Anthropic, 2 RunPod, +1. SECRET-ROTATION-RUNBOOK.md exists. Rotation must precede any git filter-repo. Never propagate creds/.acesense-worker-sa.json when replicating filesystems.
• 9 of 10 repos hold exactly one unpushed commit each (2026-07-25 batch: secret-scan workflows, CSP, abuse-limit lockdown, PII expiry). Unpushed means the hardening isn't on origin — after rotation, push all nine.
• secrets/key.json in auth-function: confirmed gitignored. Secret-scan workflows present across repos.
• Earlier auth-function OAuth token leak: history verified clean 07-25; only rotation confirmation outstanding.

6. Architecture risks, ranked

1. Client upload reliability — 13/16 recent failures. The app's chunked upload dies without completing; server correctly times it out after ~3 h. Needs app-side resumable/retry work and telemetry on chunk failure. Biggest single lever on the 69% success rate.
2. Manual filesystem deploys — one-commit drift already recreated today (se1, no capacity). Any fix is invisible to a region until someone hand-syncs it. Mitigations: a deploy script that fails loudly when any region is unreachable; a version stamp the worker heartbeats so drift is observable in Firestore.
3. Homography solver — best-quality footage returns zero value (11 videos). Highest product upside of any pipeline fix.
4. Gate asymmetry — court-present mode publishes what gated mode rejects. Either unknown shots earn their place or they don't; policy should not flip on court status alone.
5. Region capacity — 12 of 16 filesystems unseeded; se1 regularly has zero instance capacity. Seeding more regions is the availability fix; capacity-chasing at deploy time is the tax. Updated 2026-08-03: the fleet now scales to the backlog (one box per uncovered job, maxAutoInstances 6) instead of overflowing to RunPod, so unseeded regions now bound peak throughput rather than merely deploy convenience.
6. Minor: 13–15 h stale workers/ docs are cosmetic (dispatch gates on 45 s freshness) but should be TTL-cleaned; scaler ladder depends on live capacity APIs that lie sometimes.

7. Priorities

#	Action	Effort	Impact
1	Rotate 5 leaked keys (runbook), then push the 9 held hardening commits	hours	closes the standing exposure
2	Instrument + fix app chunked upload (13/16 failures)	days	success rate 69% → ~90%
3	Deploy 6182434 to us-southeast-1 when capacity returns	minutes	closes last stale region
4	Homography solver on the 11 partial videos (labelled corpus exists)	days	converts best footage from zero value
5	Unify gate evidence policy across court modes	days	consistency + fewer false publishes
6	Worker version heartbeat + loud deploy script	half-day	makes drift visible before users hit it
7	Seed remaining region filesystems	half-day each	availability ceiling

8. What is demonstrably solid

• Dispatch chain end-to-end (enqueue → claim → run → finalize → refund on failure) held through an outage, a recovery, and a corpus sweep; reconciler re-routed correctly.
• The gpu-backend test suite bites: every fix this week was mutation-tested, and two tests that didn't bite were found and replaced.
• Corpus + labels + per-video digests are now durable assets (~/Developer/caches/acesense-user-videos/, _corpus_run_2026-07-28/) — regression baseline for any pipeline change.
• Documentation now tells the truth about both deploy paths.