Skip to main content

Handoff β€” 2026-07-12

Frozen snapshot β€” do not update. Kept for how a decision was reached, not for what is true now. Current state lives in Architecture β†’ Current State.

Audience: a new engineer or agent taking over AceSense with zero prior context. Written: 2026-07-12. Backend HEAD: acesense-gpu-backend @ 4969bd2 (pushed, auto-deployed).

Read this top to bottom before touching anything. It explains what the product is, what shipped in July, what is proven vs. still open, the exact state of production and live infrastructure, and the conventions this codebase runs under. A first-hour checklist is at the end.

One thing to internalize first: the older HANDOFF.md (dated 2026-07-09) opens by saying "every Firebase deploy is blocked on expired CLI credentials." That is stale. Akshay reauthed on 2026-07-10 and Wave 1 was deployed and hash-verified to prod that day; another design-audit wave deployed 2026-07-11. Treat HANDOFF.md as a historical record of what shipped, not of what is blocked. This document supersedes it.

1. Product and repository map​

What AceSense is​

AceSense is an AI tennis-analysis product. The user flow:

Athlete uploads a match video (Flutter app)
      β”‚
      β–Ό
Firebase Functions dispatch the job β†’ RunPod GPU worker
      β”‚
      β–Ό
GPU/ML pipeline (PyTorch) extracts: shots, stroke types, pose, ball track,
      court, bounces, landings, heatmaps, per-shot clips
      β”‚
      β–Ό
Analysis JSON + PDF report + LLM coaching insights written back to Firebase
      β”‚
      β–Ό
Flutter app shows shots / overlays / insights; React admin console manages jobs

The repos (all under AcesenseProd/, each its own git repo)​

There is no umbrella repo. Each of the ~10 repos has its own .git, sits on branch main, and has remote git@github-acesense:Acesense/<name>.git. Commit per repo β€” never one cross-repo commit.

RepoStackRoleMatters most?
acesense-gpu-backendPython ~30k LOC, PyTorch, RunPodThe ML analysis pipeline. The heart of July's work.Yes
acesense-frontendFlutter/Dart ~43k LOC (Riverpod, go_router, Firebase)The athlete-facing mobile/web appYes
acesense-auth-functionNode/TS Firebase Functions ~8kCallables: admin, api, coach, feedback, video, insights (LLM + PDF)Secondary
acesense-adminReact/Vite/TS ~12kAdmin dashboard (jobs, users, moderation)Secondary
acesense-landingReact/ViteMarketing site (SEO prerender)Low
acesense-docsDocusaurus + Firebase functionsDocs site (admin-locked auth gate)Low
acesense-annotateTauri (React/TS + Rust)Internal labelling tool (App.tsx is a 1135-line god component)Low
acesense-launchpadReact/ViteInvestor/pitch site (own CLAUDE.md)Low
acesense-api-hostingFirebase hosting shimRewrite shimLow
acesense-brandbrand-book, logos, tokens/tokens.cssDesign tokens (hand-copied into 5+ consumers)Low

plan/, screenshots/, .tmp-crops/ at the product root are scratch, not repos.

Deploy model (know this cold)​

TargetHow it deploys
acesense-gpu-backendPush to main auto-deploys to RunPod. No Firebase. git push == deploy.
Everything else (frontend, functions, admin, landing)Manual Firebase CLI (firebase deploy). Project = acesense-prod.

Firebase specifics: hosting targets app β†’ acesense-prod-app (acesense-prod-app.web.app), landing β†’ acesense-prod (acesense-prod.web.app). Functions codebase is literally named functions, so single-function deploy is --only "functions:functions:<name>". All callables region europe-west1. Deploy order when multiple repos change: backend functions β†’ frontend β†’ admin β†’ landing (UIs call callables that must already exist).

2. What was done in July 2026 (newest first)​

Each item lists WHAT WORKED and WHAT DIDN'T / WATCH OUT.

2a. 2026-07-12 β€” Unified pipeline (commit 4969bd2, pushed)​

pipeline_v2 was folded into process_video as a stage between event detection and enrichment. Previously v2 ran as a post-hoc second pass that produced a separate _analysis_v2_merged.json which the cloud hook swapped in. Now there is one pipeline, one analysis JSON.

How it works (src/pipeline_v2/unified.py::apply_unified_shots):

  • Runs v2 segment-first machinery (scene segmentation β†’ gameplay-gated pose pass β†’ kink+wrist contact detection β†’ STGCN++ stroke classification) on the in-memory v1 outputs.
  • Replaces the v1 event-detector shot list with v2 contacts + stroke labels, written back in the v1 shot schema. Player mapping: near β†’ bottom / player_id 2; far β†’ top / player_id 1.
  • v1 enrichment (pose_sequence, swing_data, landing, hit_position) then runs natively on the v2 shots β€” full field parity, plus classification.{type,confidence,probabilities} (6-class) and a v2 sidecar block (detection_confidence, channel).
  • main.py cloud hook skips its legacy two-pass swap when unified shots are present.
  • Kill switch: UNIFIED_PIPELINE=0 reverts to v1 shots (and the legacy two-pass hook takes over). Fail-open: any exception leaves the v1 shots untouched.
  • A guard in shot_enrichment.py stops the legacy ONNX pose classifier from overwriting the STGCN++ label on v2 shots (checks detection_method == "pipeline_v2").

WHAT WORKED (A100-verified):

  • 1-min video: 5 v2 shots replaced 9 v1 shots, all enriched 5/5. Hook-skip produced a byte-identical JSON.
  • 5-min match: 79 shots (34 forehand / 22 serve / 18 backhand / 5 unknown); pose 78/79, swing 79/79, landing 67/79; e2e 265s.
  • Golden gate PASS; pytest 517 pass.

WHAT DIDN'T / WATCH OUT:

  • 3 pre-existing pytest failures remain (NOT caused by this work β€” verified pre-existing at HEAD via git stash): tests/insights/test_phase6_insights.py (Γ—2) and tests/pipeline/test_phase6_outputs.py::test_player_crop_rect_maps_player_id_to_top_or_bottom.

2b. 2026-07-12 β€” GPU optimization wave (commit ec2d7c6, pushed)​

The pipeline was single-core CPU-bound: aggregate CPU ~3% on 30 cores (= 1 core pegged), GPU sitting at 0–19%. Root cause: every batch was decoded/resized/stacked/transposed on the CPU strictly before the GPU ran, so preprocess and inference never overlapped.

Fix β€” new src/utils/frame_feed.py producer/consumer feeder:

  • Producer thread (CPU): decode (cv2.VideoCapture) or slice in-memory frames + cv2.resize to uint8 BGR.
  • Consumer (GPU): pinned non_blocking upload β†’ 3-frame stack + HWCβ†’CHW + float()/255 on device.
  • Numerical parity: frames stay uint8 until on-device; resize deliberately stays cv2 in the feeder (cv2's fixed-point uint8 resize is NOT bit-reproducible with F.interpolate). On-device float/255 + concat are elementwise IEEE ops β†’ bit-identical to the numpy path (torch.equal verified).

Per-stage results (samp_match_03_5min, A100-40GB):

StageBefore β†’ AfterEquivalence
Ball (TrackNet)53 β†’ 152 fpsbit-identical (dmax 0.0 px)
Players65 β†’ 239 fpsidentical on 1500 real-matrix frames
Court~1.1Γ— onlynondeterministic (see below)
v2 run_pose_pass62 β†’ 89 fpswrists/kpts/scores bit-identical
End-to-end 5-min459s β†’ 185s (2.5Γ—)all 77 shots field-for-field identical

Key findings:

  • The player-stage cost was NOT YOLO (258 fps). It was two per-frame full-res court-mask warpPerspective calls in the tracking tail (~16 ms/frame). Replaced by src/players/mask_sampling.py::warped_mask_at_points β€” point sampling via cv2.remap (same fixed-point bilinear engine), identical outputs.
  • Court model is nondeterministic run-to-run under fp16 autocast: 806/1500 frames differ old-vs-old. Never expect bit equality downstream of court β€” use a noise floor.

WHAT DIDN'T / left deliberately:

  • TrackNet batch 8 is FASTER than batch 64 on A100 (41s vs 47s). The auto_batch_size ceiling of 64 is counterproductive here. Left unchanged deliberately (unverified to change safely) β€” a candidate micro-tune.

2c. Earlier July waves (summary)​

2026-07-11 β€” pipeline_v2 built (5 parallel agents, all 27 planned improvements), merged 549afbb..3273e0b, auto-deployed. Everything lives in src/pipeline_v2/:

  • segmentation.py β€” one-pass NVDEC decode: scene cuts + dissolves + gameplay probe.
  • contact_detection.py + contracts.py β€” kink+segment+wrist contact logic, per-segment health routing, evidence-only gates, attribution, confidence.
  • classifier.py β€” STGCN++ TorchScript stroke classifier.
  • orchestrator.py, grammar.py (Viterbi rally decoder), traj_features.py (serve-variant separators), regression/golden_run.py (CI gate), cropsheets.py, flywheel.py, gemini_check.py.
  • A100-verified: golden gate PASS; tennis_1 e2e contacts exact with correct near/far attribution.

Flywheel round 1 β€” Gemini-labeled 662 contact windows across 6 videos (~86% purity). Fine-tuned STGCN++ from 12 THETIS classes β†’ 6 product classes. Holdout agreement with Gemini 20.0% β†’ 48.8% (still climbing at ep25 β†’ more labels = more gain). Shipped games/tennis/models/stgcnpp_product.pt (+ .labels.json); classifier.py prefers it when present (commit 3273e0b). Both files verified present in the repo.

Cloud auto-v2 hook (commit 891e2ee) β€” uploaded analysis JSON auto-gets pipeline_v2 applied (now largely superseded by the unified pipeline, retained as the UNIFIED_PIPELINE=0 fallback).

THETIS action-model R&D (H100) β€” PoseC3D won in-domain (73.6% subject-independent), but STGCN++ wins cross-domain (98% serve recall vs PoseC3D 58% on real US-Open pose) because its PreNormalize2D is translation/scale-invariant and travels across the broadcast/phone domain. Ship STGCN++ for real footage. See acesense-action-model-thetis.md.

Shot-contact detection R&D β€” F1 1.0 on broadcast; ~0.87–0.93 verified precision across 6 videos / 4 domains. Ball-centered crop sheets are the verification standard (contact happens where the ball is). See acesense-shot-detection-kink.md.

Design-audit wave (2026-07-11) β€” 20+ UI fixes across 4 repos, deployed same day. See acesense-design-audit-wave.md.

Insights + PDF moved to a Cloud Function β€” generateSessionInsights in acesense-auth-function/insights/. Gemini AQ.-format key works only via REST generativelanguage.googleapis.com/.../generateContent with the X-goog-api-key header β€” the @google/genai SDK 401s it. Use fetch, not the SDK. See acesense-insights-report-function.md.

3. What did NOT work / known failure modes​

Be honest with these; they cost real time and will bite again.

Environment / dependency traps​

  • decord segfaults on the GPU boxes β†’ read frames with cv2, never decord.
  • mmpose is blocked by chumpy (setup.py does import pip). Use ultralytics YOLO11x-pose for pose extraction instead.
  • onnxruntime-gpu pulls a CUDA-13 libcudart (boxes are CUDA 12.8). Avoid it.
  • numpy must be pinned 1.26.4 LAST with --ignore-installed β€” mmaction/other installs re-bump it to 2.x and break ABI.
  • scipy 1.15 wheels break with numpy 1.26 on py3.10 β†’ use scipy 1.13.1.
  • Lambda boxes: system pip has an invalid flatbuffers distro version that breaks pip; a venv --system-site-packages produces ABI soup (scipy/numpy _fitpack TypeError). Always use an isolated venv. System ffmpeg has NO CUDA hwaccel.

SSH gotchas​

  • pkill -f <pattern> over ssh: if <pattern> appears anywhere in the remote command line (including later nohup args), it kills its own session. Split into separate ssh calls.
  • Multiline python -c '...' inside ssh single-quotes hangs. Always scp a script file and run it.

Algorithm / domain failure modes​

  • Homography explodes to Β±300m on court-level footage exactly at contacts. Physics-bounds hygiene exists (drop court samples outside x[-6,17] y[-8,32], cap velocity 60 m/s).
  • court_y depth is garbage on court-level (behind-baseline) footage β€” the homography's ill-conditioned axis. Both detectors go mediocre there. Production fix path: image-space ball kinematics for near-player contacts + audio onset detection (geometry-free ~10 ms).
  • Gameplay gate needs court_frames β‰₯ 100 to activate, else it nukes court-level recall to 0 (the court detector barely fires from behind the baseline, so the gate must self-disable there).
  • Scene-cut gate must NEVER touch ball-classified hits β€” a 6Οƒ "cut" on broadcast was a fast pan and deleted a real hit. Gate wrist-adds only, and reject only on positive evidence (ball tracked AND far), never on missing data.
  • Residual known false-positive class = ball-juggle-while-walking β€” kinematically inseparable from a soft feed; belongs to rally/point segmentation, not kinematics.
  • THETIS "skeleton" folders are rendered videos, not joint data β€” extract your own pose.
  • Published THETIS numbers leak subjects (3 near-identical reps per player). Always evaluate leave-players-out (subject-independent).

4. Current production state​

ThingState
acesense-gpu-backendmain @ 4969bd2 β€” auto-deployed to RunPod on push. Working tree clean.
Unified pipelineDefault ON. Kill switch UNIFIED_PIPELINE=0. Fail-open to v1 shots.
FrontendNeeds NO changes. The model is defensively nullable and reads classification.type etc.; unknown/missing fields degrade gracefully.
Firebase deploysNot blocked. The 2026-07-09 expired-CLI-creds blocker was resolved 2026-07-10; Wave 1 + the design-audit wave are live and hash-verified. (Re-check acesense-fix-sweep-pending-deploy.md if creds seem stale again.)
Deployed URLsapp acesense-prod-app.web.app, admin acesense-admin.web.app, landing acesense-prod.web.app, health https://health-p2byrhfnxq-ew.a.run.app.

Local artifacts store (canonical, not in any git repo): ~/Desktop/Workspace/RunningProjects/acesense/tennis-action-artifacts/ Contains: checkpoints/ (exported models), shot-detection/ (the ONLY canonical copy of the shot-detection R&D scripts β€” a scratchpad wipe once destroyed them, so keep patches here), pipeline-v2-verify/ (labels.jsonl = 662 Gemini labels, finetune_product.py, flywheel_label.py, verification evidence, stgcnpp_product.pt).

5. Live infrastructure RIGHT NOW (costs money)​

A Lambda A100 box is still running with the full rig:

  • Host: ubuntu@132.226.88.61 (ssh key ~/.ssh/id_ed25519_lambda)
  • ~/gpu-backend β€” synced working tree (post-optimization)
  • ~/gpu-backend-baseline β€” pre-optimization tree (for equivalence gating)
  • ~/venv β€” isolated: torch 2.7.0+cu126, numpy 1.26.4, scipy 1.13.1, ultralytics, mediapipe 0.10.5
  • ~/videos_tmp_dir/ β€” tennis_1.mp4, samp_match_02_1min.mp4, samp_match_03_5min.mp4
  • ~/prof/ β€” all equivalence/verification logs + scripts

This box costs money. Terminate it via the Lambda API when it is no longer needed β€” this is Akshay's decision, not an automatic one. Do not terminate without confirming.

6. Open items / next steps​

ItemNotes
Flywheel crank 2More Gemini labels β†’ retrain STGCN++ toward 90%+ holdout agreement. Tools: finetune_product.py + flywheel_label.py in tennis-action-artifacts/pipeline-v2-verify/. Current agreement is 48.8% and was still climbing β€” this is the biggest accuracy lever.
Watch first real cloud jobTail the first real RunPod cloud job logs for the unified-stage line unified: v2 shots active. Confirms the unified pipeline fires in prod, not just on the A100.
Frontend surfacing (optional)Consider showing v2 detection confidence + the 6-class probabilities in the UI. Backend already emits them (classification.probabilities, v2.detection_confidence).
TrackNet batch ceilingMicro-tune the auto_batch_size ceiling (8 vs 64 on A100). Unverified β€” gate it.
e2e tailRemaining time is video loading (~18s) + output generation (~30s) + enrichment (~25s). Diminishing returns; only chase if needed.
Landing legal factsRegistered entity name + address, Delaware-Inc vs "EU company", "no data leaves the EU" vs US-servers+SCC. Founder-decision, agent must not invent. See HANDOFF.md Β§6.
Wave 2 backlogNotifications (push needs APNs/VAPID keys), storage retention, App Check (needs reCAPTCHA v3 key), full l10n. See HANDOFF.md Β§7.
PaymentsExplicitly excluded by Akshay. Do not implement IAP/payments.

7. Working conventions (non-negotiable)​

From acesense-working-style.md β€” "Ponytail mode":

  • Boring, small, verified changes. Prefer the smallest cut that satisfies the item. No new frameworks/DSLs/codegen unless a smaller fix genuinely can't work. Do not convert to a monorepo.
  • One commit per repo per logical change. Never a cross-repo commit. Push only when asked.
  • Every optimization is equivalence-gated: run OLD twice first for the noise floor, then compare old-vs-new. Court/anything downstream of it is nondeterministic β€” use the noise floor, never expect bit equality.
  • Ball-centered crop sheets are the visual-verification standard for detections.
  • Never fabricate. Real numbers/identity/scores or an honest empty state β€” this is a hard product rule (kUseMockData is the only mock path; injury//body content must never make medical claims).
  • Never fabricate test runners. Backend uses pytest; frontend uses Flutter (flutter analyze + flutter test); TS repos use pnpm + vitest. There is no pnpm lint on the backend, etc. β€” use what each repo actually defines.
  • Agents never git push without the verification battery passing first (golden_run + pytest on backend).

Reference material (don't duplicate β€” read these)​

  • src/pipeline_v2/CONTRACT.md β€” the module interface contract every v2 module obeys (fps-invariance, evidence-only gates, numpy+cv2+stdlib only in core, torch allowed only in classifier.py).
  • HANDOFF.md β€” the 2026-07-09 Firebase/frontend/admin/landing wave detail: exact field mappings, deploy sequence, decisions. Still-valid frontend field mapping and deploy mechanics live there.
  • acesense-local-ui-rig.md (memory) β€” how to run all 3 UIs against the Firebase emulator and screenshot them headlessly, incl. emulator ports (auth 9099 / firestore 8080 / functions 5001 / storage 9199 / UI 4000), demo creds (athlete@demo.acesense.io / Demo1234!, admin admin@acesense.io), and the headless-auth injection tricks.
  • The memory directory (~/.claude/projects/-Users-akshaysarode-Desktop-Workspace-RunningProjects-acesense-AcesenseProd/memory/) β€” MEMORY.md is the index; the per-topic files are the deep record.

8. First-hour checklist for the new agent​

  1. Read src/pipeline_v2/CONTRACT.md in acesense-gpu-backend β€” it defines the shared types and the rules every v2 module follows. This is the single most important orientation file for the ML work.
  2. Run the backend test suite: pytest in acesense-gpu-backend. Expect 517 pass, 3 pre-existing failures (2Γ— tests/insights/test_phase6_insights.py, 1Γ— test_phase6_outputs.py::test_player_crop_rect_maps_player_id_to_top_or_bottom). Anything beyond those three is a regression you introduced.
  3. Run the golden CI gate: python -m src.pipeline_v2.regression.golden_run β€” it exits nonzero if tennis_1 F1 < 1.0 or rally F1 drops below baseline. This is the shot-detection safety net.
  4. Check the RunPod dashboard for a real cloud job, and grep its logs for unified: v2 shots active to confirm the unified pipeline is firing in production (not just on the A100).
  5. Decide the fate of the Lambda A100 box (ubuntu@132.226.88.61) β€” it is running and costing money. Confirm with Akshay before terminating; terminate via the Lambda API when the verification rig is no longer needed.

End of handoff. When in doubt, prefer reading the memory files and CONTRACT.md over inferring from code, and keep changes boring, small, and equivalence-gated.