Skip to main content

Current State

Updated: 2026-08-17. This is the current orientation doc. Everything in §3 was verified by running the check on that date, not carried forward from notes; anything measured earlier is labelled with when.

Older handoffs, audits and briefs live in the Archive and are frozen snapshots. Do not update them — read them for how we got here, not for what is true now.

1. Product (short)​

AceSense = AI racket-sport analysis.

Flutter upload → Firebase Functions → GPU worker → analysis JSON + clips + insights → app

GPU/ML heart: acesense-gpu-backend (PyTorch). Capacity: the Lambda pull-worker fleet is primary — lambdaScalerTick launches one box per uncovered queued job up to maxAutoInstances (Firestore config/gpuScaler), across a region ladder. RunPod is fallback only and can run out of credit; it is not the capacity plan. Multi-provider dispatch lives in acesense-auth-function/video/gpu/.

Instances are named acesense-worker-auto-*. The sibling UNFORCE product shares the same Lambda account and gcloud environment, so the prefix is load-bearing: orphan reclaim scopes on it, and anything named unforce-* must never be touched. gcloud config does not default to acesense-prod — pin --project acesense-prod in every script.

Repos under AcesenseProd/ are separate git repos (no umbrella). Commit per repo. Remotes: git@github-acesense:Acesense/<name>.git.

RepoRole
acesense-gpu-backendML pipeline (critical)
acesense-frontendFlutter athlete app (web + mobile)
acesense-auth-functionFirebase callables + GPU dispatch
acesense-admin / landing / docs / launchpad / annotate / brand / api-hostingSecondary

2. Pipeline (what runs)​

Unified pipeline is default ON (UNIFIED_PIPELINE=1). Kill switch =0 → legacy v1 + post-hoc v2 swap.

Flow (sequential, 1 GPU):

  1. Video load (dual-res: original + 640×360 model)
  2. Court (CourtNet, every 10th frame)
  3. Ball (TrackNet, all frames, batch capped at 8)
  4. Players (YOLO11m, all frames @ original res)
  5. Bounce / events
  6. Unified v2: segmentation → gameplay-gated pose (YOLO11x-pose @960) → contacts → STGCN++
  7. Enrichment (MediaPipe / swing; ONNX classifier skipped for v2 shots)
  8. Outputs (JSON + shot dirs; clip ffmpeg on hot path)

Pose already skips non-gameplay. Ball + players do not — segmentation runs too late to gate them.

Conventions: boring/small/verified changes; equivalence-gate optimizations; pytest on backend; never fabricate scores or identity.

3. Current state — verified 2026-08-17​

Repos and CI​

All repos: dirty=0 unpushed=0 untracked=0, all on main.

CI runs locally — GitHub Actions is retired for this org (2026-08-24). Every gate the workflows used to run now lives in ci-local.sh at the family root: per-repo lint/tests/coverage/build, the brand-separation gate (ported into each repo as .ci/brand-separation.sh), and the gitleaks secret scan of a HEAD export. Opt-in modes replace the scheduled/manual workflows: FULL_HISTORY=1 (all-history secret scan), MUTATION=1 (auth-function mutation gate), LOAD_SMOKE=1 (k6), RULES_EMULATOR=1 (Firestore rules). Run it before every push; a red sweep is a red pipeline.

One deliberate standing red under FULL_HISTORY=1:

  • acesense-gpu-backend → the pushed history contains a production GCP service-account key, an Anthropic key and two RunPod keys. The scan is correct. Rotate before rewriting history — scrubbing first leaves live credentials in circulation with no record of where they were. unforce-gpu-backend's history was rewritten clean on 2026-08-16. See Secret Rotation.

Deploy — read this before shipping anything​

All deploys are manual and local. The hosting-only deploy workflow that used to run on v* tags (and failed every time for lack of credentials, leaving the live build four versions behind main for two weeks) is gone with the rest of GitHub Actions. Consequences:

  • Git tags are release markers, not deploy triggers.
  • Web deploys: cd acesense-frontend && ./deploy.sh (needs APPCHECK_RECAPTCHA_KEY — a public reCAPTCHA v3 site key, not a secret; it is baked into the served bundle and recoverable from it).
  • App Check service enforcement: ./scripts/enforce-app-check.sh ENFORCE_SERVICES in the frontend repo, after service metrics are healthy.
  • Run ./ci-local.sh green before any deploy.

What is live​

All hosting surfaces were deployed and verified on 2026-08-16 and tagged deploy-2026-08-16 in their repos (date-based, because their package.json versions are placeholders). Admin and landing were both stale before that — the served asset hash did not match a fresh build.

SurfaceState
app.acesense.io / acesense-prod-app1.0.22, tag v1.0.22, CSP enforcing, 0 console errors
acesense.io / acesense-prod.web.appLanding, current. CSP report-only, never load-tested
acesense-admin.web.appAdmin, current. CSP report-only, never load-tested
acesense-launchpadHosting + functions:launchpad, current. 401 gate intact
acesense-prod-docs401, auth-gated (expected)
acesense-prod-api.web.appRewrite host, CSP enforcing
Firebase Functions (functions codebase)64 deployed, current as of 2026-08-16
Lambda GPU fleetScales to zero. Boxes launch per uncovered queued job and are reaped when idle — verified 2026-08-17 with a live job: one tracked acesense-worker-auto-* launched, no orphan
acesense-annotateTauri desktop app — nothing to deploy

The table records the last live verification. Source configs for landing and admin moved to enforced CSP on 2026-08-21, but those changes have not been deployed or checked through the custom-domain/CDN layer yet.

Functions are current. Deployed 2026-08-16, tag deploy-2026-08-16. The per-box runtime budget and the acesense-worker-auto-* rename are running; they had been stuck since 2026-08-03. retryInfraFailedJobs and reconcileStuckJobs were redeployed 2026-08-17.

The ffmpeg -fps_mode probe is not a Functions change and was not live with them — it ships in acesense-gpu-backend, which deploys separately. See below.

Worker code does not deploy with anything else​

Scaler-launched boxes clone nothing. Each Lambda region has its own acesense-worker filesystem holding a plain copy of acesense-gpu-backend, and the instance mounts it and runs bootstrap.sh. There is no git pull and no CI step, so a merged commit is not running anywhere until it is copied onto every region filesystem — and nothing reports the drift.

Measured 2026-08-17, every filesystem reachable that day was the same four files behind main: vfr_normalizer.py, video_encoding.py, chunked_runner.py and tennis_analysis.py — i.e. the ffmpeg frame-rate-mode fix (c046848, d1d1edd, merged 2026-08-14) had never reached a worker anywhere. File counts matched exactly, 121 either side with nothing missing or extra, so the drift is purely un-shipped commits rather than local edits.

Verify a drop by checksum, not by assumption — there is no deployed-commit marker on the filesystem and no .codedrop_head in the repo:

ssh …@<box> 'cd ~/acesense-worker/acesense-gpu-backend && find src games worker main.py requirements -type f \( -name "*.py" -o -name "*.sh" -o -name "*.txt" \) | sort | xargs md5sum'

A missing asenv/bin/python on a freshly launched box is expected, not a broken region: the venv's interpreter is a symlink to /usr/bin/python3.10, which bootstrap.sh apt-installs, and a maintenance box that never ran bootstrap leaves it dangling.

RegionFilesystem state
us-east-1Current — 166/166 files checksum-identical to main, 2026-08-20
us-west-1Current — 166/166, 2026-08-20
us-southeast-1Current — 166/166, 2026-08-20
us-south-2Current — 166/166, 2026-08-20

That drop carried the five-module package split, so the file count moved from 121 to 166. It was applied with acesense-gpu-worker/scripts/fleet_sync.py, which refuses to run while the control-plane snapshot has drifted from the deployable repos, and verifies creds/ and asenv survive before reporting success.

Two transport notes worth keeping, both measured on 2026-08-20:

  • The cloudflared tunnel throttles to roughly 1.7 KB/s. rsync sends only differences so a normal drop is a minute or two, but a first-time region seed over this link is not viable — and never tar games/, which is 122 MB of model weights and would take a day.
  • Lambda recycles IPs. A maintenance box can come up on an address already in known_hosts with a different key, and fleet_sync uses StrictHostKeyChecking=accept-new, which fails rather than replaces. Clear it with ssh-keygen -R <ip> before blaming the tunnel.

us-south-2 appears in the scaler ladder — including the only 2x rung — so a job placed there still runs the pre-2026-08-17 pipeline and can still hit the GPU_REJECTED tolerance bug. Copy the four files when capacity returns; it had none all day on 2026-08-17, while us-southeast-1 went from none to available within the hour, so re-check rather than assuming.

Updating a filesystem means launching a box in that region that mounts it. Prefix maintenance boxes acesense-maint- so the orphan reclaim, which is scoped to acesense-worker-auto-*, leaves them alone — and terminate them. Copy individual files rather than rsync --delete: the filesystem also holds creds/.acesense-worker-sa.json and the asenv virtualenv, and deleting either takes the whole region's worker launches down.

APP_STORE_PRIVATE_KEY no longer blocks anything. Apple billing sits behind ENABLE_APPLE_BILLING, off by default, so verifyApplePurchase and appStoreServerNotification are not exported and the secret is never declared. Neither had ever been deployed. Android store billing (verifyGooglePlayPurchase, googlePlayBillingNotification) went live in the same deploy and needs no secret — it authenticates with ADC.

The trap worth remembering: gating the exports was not enough. defineSecret() and defineString() register a parameter with the deploy even when nothing exported consumes them, so the declarations themselves have to be conditional. Two unshipped iOS entry points had held back 17 commits of entirely unrelated work.

To enable iOS: set APP_STORE_PRIVATE_KEY in Secret Manager, then deploy with ENABLE_APPLE_BILLING=true.

Functions cannot be partially deployed around the missing secret — Firebase resolves secrets while analysing the whole codebase, so even a scaler-only deploy aborts.

Frontend health​

flutter analyze clean · 512 tests pass · dart format clean · tool/l10n_audit.py reports 0 hardcoded user-facing strings · firestore.indexes.json has 0 drift against production.

Localisation — done 2026-08-16​

Six locales (en de es fr it sv), 1542 keys each, no drift. The reported bug ("still in English after switching") was two independent defects:

  1. ~280 literals never looked up at all, across 33 files.
  2. ~200 keys wired correctly but carrying the English string as their non-English value — Badges, Roadmap, the Profile nav and Settings rendered fully English even though every call site was right.

Three catalogues cached copy behind a provider, so words resolved at build time outlived a language switch. PlayerCoachingService, DrillItem and ShotType now carry facts and stable ids; the widget layer turns them into words per render. ShotType.stableName exists specifically because one lookup compared against the English label — localising it would have silently broken that comparison in five locales.

tool/l10n_audit.py gates this in CI (between analyze and test). It is a net, not a proof: it cannot see a literal reaching the widget through an intermediate parameter — that is how Good morning survived it. Render in German and read the screen. See §5.

3a. Pipeline health (2026-08-17)​

CI being green says the code compiles and the tests pass. It says nothing about whether real uploads succeed. On 2026-08-16, 40 of the last 60 jobs had failed. Four distinct causes, all now understood:

errorCodeStatus
GPU_JOB_FAILEDFixed. RunPod removed; 14 stranded jobs auto-recovered
CLIENT_UPLOAD_FAILEDFixed in code, needs a Play release to reach users
STUCK_JOB_TIMEOUTMostly a genuinely abandoned upload — but not always, see below
GPU_REJECTEDWas not the clip's fault. Fixed 2026-08-17, see below

The failure counts, measured against Storage (2026-08-17)​

Two of the four rows above were assumptions, and checking each failed job's videoPath against the bucket disproved both. Over the preceding 60 days:

errorCodenvideo still in Storage
GPU_REJECTED1313
STUCK_JOB_TIMEOUT363
CLIENT_UPLOAD_FAILED402
GPU_JOB_FAILED11

"The upload never arrived" is right for most STUCK_JOB_TIMEOUT and CLIENT_UPLOAD_FAILED jobs and wrong for a few, and it was never right for GPU_REJECTED at all.

GPU_REJECTED — integrity tolerances in the wrong units​

All 13 were healthy videos. The CFR integrity check compared container duration against frames / fps with a tolerance of frame_period * 0.002 — 67 µs at 30 fps. Container duration is stored in time-base ticks, and Matroska (what the chunked path writes) quantizes to 1 ms, so the bound sat well inside a single tick and measured rounding rather than damage. Observed deltas ran 67 µs to 2.55 ms, the largest a seventeen-hundredth of one frame. One user re-uploaded the same clip three times and was refused three times.

That bound also contradicted the frame-count check beside it, which forgives a couple of frames of VFR→CFR resample drift — tens of milliseconds of duration, a thousand times the duration bound. CFR only ever passed because the relaxed retry path re-validates without expected_duration_s at all.

Both bounds now derive from one allowance so they cannot disagree, with the self-consistency bound held strictly under one frame period (phones shoot slow motion at 120 and 240 fps, where a frame is shorter than a flat tick floor). All 13 production rejections are pinned as regression tests, along with negative controls — a second of truncation is still 4× the budget.

STUCK_JOB_TIMEOUT — one was a lost Storage event​

Job rCl1eJ0lGcBoMV6hFyH6 sat pending with a complete 661 MB video in the bucket, no uploadedAt and no preflight: processVideoOnUpload never fired for it. Storage events are at-least-once, not exactly-once. The sweeper failed it and told the user their upload never completed — destroying a finished upload rather than recovering one.

reconcileStuckJobs now checks Storage before failing an overdue pending job and re-fires the finalize event when the object is there. That job itself is already failed and needs a manual retry; the fix prevents the next one.

CLIENT_UPLOAD_FAILED — "Bad state: Stream has already been listened to"​

FileDownloader().updates is backed by a plain StreamController in the plugin, not a broadcast one, so it accepts exactly one listener for the life of the isolate. background_upload_io.dart subscribed per upload: the first upload of a session worked and every one after it threw immediately — including the user's own retries, so a failed upload stayed failed until the app was restarted. One user hit it five times in four minutes on a 1.5 GB clip.

Fixed by a single permanent listener feeding a broadcast controller. Cancelling does not help: re-listening to a single-subscription stream throws even after a clean cancel, so the source subscription is deliberately never cancelled.

io-only, so it reaches users through a Play release. The web build was never affected. Until that release ships, mobile users still hit it.

GPU_JOB_FAILED — RunPod removed, and jobs now self-heal​

RunPod was a paid fallback sitting at 402 Insufficient Balance, so a dead Lambda worker did not mean slower, it meant the job died with the user's video intact in Storage. Three places routed to it, including DEFAULT_GPU_PROVIDER_POLICY — the policy that applies if GPU_PROVIDER_POLICY is unset or malformed, so fixing only .env would have left a config slip able to resurrect a dead provider. All three are Lambda-only; tests pin it.

lambda-reconcile now requeues a dead worker's job onto the Lambda queue, refreshing the signed URL in the queue doc — it is minted per dispatch and may have expired while the worker was dying, and a requeue handing the next agent a stale URL fails silently.

retryInfraFailedJobs (every 10 min) re-dispatches jobs that failed for infrastructure reasons and whose video still exists. It recovered all 14 stranded jobs. Bounded on purpose: only GPU_JOB_FAILED, capped per job and per tick, skips jobs over a week old, and refreshes createdAt with a server timestamp because the stuck-job sweeper fails any active job older than ten minutes.

The GPU cost leak — a failed bootstrap made a box immortal​

Two 2×H100 boxes billed 19 and 25 hours doing nothing, and idleTerminateMinutes: 5 was correct config that could not fire.

The scaler evaluated bootstrap first and did bootingCount++; continue; on failure. A box whose ssh bootstrap could never succeed was therefore counted as "still booting" forever and never reached boot-timeout, the lifetime cap, or idle-terminate. The scaler logged bootstrap_failed on every one-minute tick and acted on none of them.

Boot-timeout now runs before bootstrap, and the ordering lives in a pure instanceVerdict() with the immortal case pinned by a test. Stale heartbeat docs are also collected — seven had accumulated, the oldest 13.6 days.

Verified after the fix with real traffic: a queued job launched exactly one tracked box on the first ladder rung. Tracking is the part that had been missing — an untracked box is invisible to every reaper rule.

Still open: the orphan reclaim is prefix-scoped to acesense-worker-auto-*, so a legacy bare worker-auto-* box is invisible to it and must be killed by hand. That is deliberate — the Lambda account is shared with UNFORCE and a name that merely looks like ours is not ours — but it is a gap.

Sign in with Apple — removed 2026-08-17​

The button is gone from sign-in and sign-up, gated behind kAppleSignInEnabled (a bool.fromEnvironment const, so the branch tree-shakes out of release builds). AuthService.signInWithApple is untouched — re-enabling is one --dart-define.

The four Apple-only accounts were deleted at the owner's instruction, with the deleteUser cascade removing jobs, coach data, API keys and the videos//results/ Storage prefixes. All four had zero jobs and zero files — signed up in May, signed in once or twice, never uploaded — so no user content was destroyed. Provider mix is now google.com 352, password 131, apple.com 0.

Worker bootstrap and the CUDA guard​

Boxes were failing bootstrap with

InferenceProviderError: ONNX Runtime silently fell back from CUDA
(active providers: ['CPUExecutionProvider'])

That guard is correct — src/utils/onnx_inference.py refuses to start a worker that would run inference on CPU, which would be silently slow and useless. The scaler logged bootstrap_failed every tick and (before the reaper fix) let the box bill forever.

Investigated on 2026-08-17 by SSH onto boxes mounting the us-south-2 acesense-worker filesystem. The venv is healthy and the failure did not reproduce:

  • onnxruntime-gpu==1.20.2 installed, CUDAExecutionProvider available
  • nvidia-cudnn-cu12 9.1.0.70 present, all eight libcudnn*.so.9 load
  • validate_cuda_onnx_runtime() returns ('CUDAExecutionProvider', 'CPUExecutionProvider') on both gpu_1x_h100_sxm5 and gpu_2x_h100_sxm5, so it is not GPU-count specific
  • bootstrap.sh completes with BOOTSTRAP_OK and the agent runs

One real trap was confirmed along the way: loading libonnxruntime_providers_cuda.so directly, without the preload, fails with libcudnn.so.9: cannot open shared object file. The libraries live in the nvidia-cudnn-cu12 wheel and are not on the default loader path, so _preload_pytorch_cudnn() in onnx_inference.py is load-bearing — it is what makes CUDA work at all here, and it is gated behind import torch succeeding.

Not reproduced, so not fixed. The failures were in an 08:00–10:26 window and the same code passes now. If it recurs, the reaper limits it to one boot window of spend rather than a day, and the ONNX error text is in the bootstrap_failed log line.

Untested: us-east-1. Each region has its own copy of the filesystem, and the first failure was on a gpu_1x_a100_sxm4 there. Only us-south-2 was verified.

Notification copy​

Recovered jobs were sending the first-time "your report is ready" mail to users whose previous email about that same job said it had FAILED. reanalyzed was keyed on backfilledAt, which no retry path sets; it now covers backfill, admin retry and infrastructure retry via isReanalysis().

4. Sport scope​

AceSense is tennis only. acesense-gpu-backend/games/ contains exactly one directory, tennis, and there is no per-sport dispatch to add a second one to. Nothing in this product accepts, routes or reports on another racket sport.

:::note Corrected 2026-08-19 This section previously carried a five-row table — tennis, badminton, padel, pickleball, table tennis — with a per-sport verdict for each. That table describes UNFORCE, the sibling product, and was carried across when this repo was forked. It was wrong here from the day it landed, and it sat three sections after a paragraph promising everything had been verified by running the check, which is the worst place for a stale claim to hide.

If you want the five-sport picture, it lives in the UNFORCE docs. Do not read it as a roadmap for this product; adding a sport here is a backend project, not a configuration change. :::

The tennis findings from that work do carry over, because tennis is the sport they were measured on:

SportE2ESolves courtNames strokesVerdict
TennisyespartialyesComplete product. Court solve is the weak link

The upstream defect is homography, not detector recall. Eleven clips marked "partial" see the court in 79–100% of frames but solve 0–1 of them. (That count comes from the shared 64-video corpus run, which predates the fork and covers more than tennis — the mechanism is the same, the absolute number is not AceSense-only.) The detector finds keypoints; the solve rejects them. This matters because court.health.status flips the gate policy wholesale — "full" bypasses the stroke gate entirely, "partial" applies it in full. Two opposite failure modes hanging off one field. Fixing the solve is worth more than any detector tuning.

Roughly half of recorded "failures" are uploads that never completed, not analyses that failed. Separate those before reading the failure rate as a model problem.

5. Traps that have already cost time​

  • Flutter goldens are Linux-only. test/preview/goldens/*.png are rendered by CI on ubuntu. Regenerating them on macOS swaps CoreText for FreeType and blows the 3% CI tolerance — 37 red goldens with no UI change. Local runs pass either way (12% off-CI), so it only appears after a push.
  • A quiet browser console proves nothing. A CSP header change does not reach a warm tab; the browser replays the cached document with its old headers. Verify in a fresh tab on a cache-busted URL.
  • Failed to fetch is ambiguous between CORS and CSP. When probing a policy, always include a known-disallowed host as a negative control — without it, "no violations" is indistinguishable from "the listener never fired". This is what caught connect-src blocking Firebase Auth's authDomain and breaking social sign-in after the CSP went enforcing.
  • Firestore appends __name__ implicitly. A naive diff of live indexes against firestore.indexes.json reports false positives.
  • firebase deploy --only functions:NAME aborts — the codebase is named functions; use --only functions.

6. Open​

ItemNotes
Scrub gpu-backend historyCredentials were rotated 2026-08-17, so the exposed keys are now dead. The full-history scan stays red until the history is rewritten; that is now a cleanup, not an incident
Play releaseCLIENT_UPLOAD_FAILED is fixed in code and ships manually; mobile users keep hitting it until that build lands (§3a)
Court homography solveHighest-value product fix (§4)
Landing + admin CSPsReport-only, never load-tested. Run the probe in Security before enforcing
Legacy-prefix orphan reclaimA bare worker-auto-* box is invisible to the prefix-scoped sweep and must be killed by hand (§3a)
Privacy copy names AppleStill lists Apple as a sub-processor for Sign in with Apple. Legal text — needs outside input
Flywheel / STGCN++Still the biggest accuracy lever

7. Pointers​

  • Pipeline contract: acesense-gpu-backend/src/pipeline_v2/CONTRACT.md
  • Bench numbers + approved quotas: GPU Benchmarks
  • Speed plan: Speed Roadmap
  • Secret rotation: Secret Rotation
  • GPU dispatch: acesense-auth-function/video/gpu/
  • Web deploy: acesense-frontend/docs/DEPLOYMENT.md
  • l10n gate: acesense-frontend/tool/l10n_audit.py
  • History (do not treat as live): the Archive