Speed Roadmap

Goal: drive wall time toward ~⅓ of video duration (5‑min clip → ~100s), using one GPU, primarily ml.g5.4xlarge (A10G). Multi-GPU is explicitly out of scope until this list is exhausted.

Baselines (SM, samp_match_03_5min):

Setup	Wall	Gap to 100s
A10G g5.4xlarge v2 cpuopt	430s	4.3×
A10G g5.16xlarge v1	443s	4.4×
A100 p4d.24xlarge v1 (1 GPU used)	331s	3.3×
L4 g6.4xlarge v1/v2	~486–494s	~5×

Hardware alone will not hit 100s. g5.16xlarge ≈ g5.4xlarge. A100 helps (~23% vs A10G) but still needs the same code work.

---

What the runs taught us

1. GPU-bound, not CPU-bound for wall. High-CPU SM SKUs (8xlarge / 16xlarge) did not move L4/T4/A10G walls. Court already uses ~12 threads; ball/pose dominate on the GPU.
2. A10G > L4 > T4 on this pipeline. Prefer ml.g5.4xlarge as the optimisation target SKU.
3. cpuopt v2 helped A10G mainly on ball (135→96s) and enrich (27→12s). Pose unchanged (~111s) — still yolo11x @960. L4 saw little net gain.
4. Video load got slower on v2 SM (~14→41s) while ball got faster — worth verifying NVDEC is actually used in the SM container (not falling back to OpenCV) and that dual-res materialization is still the bottleneck.
5. Pose already skips non-gameplay; ball + players do not. Segmentation runs after full-frame ball/players → largest unused cut (~½ frames on this clip).
6. Players run on full-resolution originals — expensive vs model-res 640×360.
7. TrackNet batch >8 is not free speed (historically slower on A100); keep 8 unless re-proven on A10G.
8. EC2 at G=8 is irrelevant for this goal (32 GB thrash). Stay on SM.
9. p4d is a faster single-GPU today only because A100 is faster — not because of 8 GPUs.

---

Recommended work (1×A10G only)

Tier A — low risk, days (expect ~430s → ~280–340s)

#	Action	Stage hit	Evidence / note
A1	ACESENSE_POSE_IMGSZ=640	pose ~111s	Env already wired; measure quality
A2	Swap yolo11x-pose → yolo11s-pose (or m)	pose	Largest remaining GPU stage on A10G
A3	Raise ACESENSE_POSE_BATCH to 24–32 on A10G	pose	24 GB VRAM headroom
A4	Players: yolo11n/s + infer on frames_model	players ~48s	Config + game_processor call site
A5	Court SAMPLE_INTERVAL 10→20	court ~63s	Sparse already
A6	Skip MediaPipe entirely for pipeline_v2 shots	enrich	Today early-return only skips ONNX; MediaPipe still ran before that
A7	Confirm NVDEC active in SM; fix fallback spam	video	v2 video_s regression

Ship each behind env flags; equivalence-gate shot counts / stroke mix on samp_match_03_5min.

Tier B — architecture, 1–2 weeks (expect ~150–200s stacked with A)

#	Action	Why
B1	Run segmentation first; pass gameplay spans into ball + players (frame_range already on BallDetector)	Unlocks ~1.5–2× on ball+players without new models
B2	Stream model frames (path/FrameFeed); stop holding full dual-res list for infer	Cuts RAM + CPU starve; SM ball was heavy when host busy
B3	Defer shot-clip ffmpeg off synchronous wall	Phase-6 multi-clip encode is silent wall time
B4	Overlap court post with early ball GPU where safe	Court infer is sparse; post is threaded already
B5	TensorRT (or compile) TrackNet + CourtNet + pose	Steady stage gains after algorithmic cuts

Tier C — product tradeoffs (needed for reliable ≤100–130s)

#	Action	Tradeoff
C1	Ball every-2nd-frame + interpolate	Trajectory smoothness
C2	Pose only near contacts, not full gameplay	Wrist/channel recall
C3	Drop enrich / swing MediaPipe in prod path	UX fields

Honest stack: A+B on A10G → likely ~150–200s. A+B+C → ~100–130s zone. Multi-GPU only after this.

---

Suggested experiment order (SM)

1. Re-bench ml.g5.4xlarge after each Tier A flag (same RUN naming *-optA1, …).
2. Implement B1 gameplay gate — single biggest structural win; re-bench.
3. B2 + B3; compare wall and video_s/outputs.
4. Only then consider A100/p4d as a faster card for the same code — still 1 GPU.

Compare script / S3 layout: see benchmark.md.

---

Explicitly defer

• Multi-GPU (g5.12xlarge, p4d 8-way)
• More EC2 / EU SM training (no approved capacity that fits)
• Raising ball batch above 8 without a dedicated A10G sweep
• “Just use H100/Lambda” for this SM plan