GPU Benchmarks
Date focus: 2026-07-13 (plus Vertex benches from 2026-07-12).
Workload: samp_match_03_5min (~5 min video), unified pipeline ON, ACESENSE_BALL_BATCH=8.
Account (AWS): 109375139591. GCP project: acesense-prod.
Only approved / currently usable quotas are listed in §2. Pending requests are omitted.
1. Pipeline versions under test
| Version ID | Image / code | What changed vs prior |
|---|---|---|
| v0 — Vertex week | GAR …/pipeline:bench-5ba30d6-fix1 | Unified pipeline + FrameFeed-era backend used for Vertex L4/T4/V100 |
| v1 — SM/EC2 baseline | ECR acesense-gpu:bench-5ba30d6-fix1 (+ -sm entrypoint overlay) | Same app code as v0; SageMaker/EC2 entrypoints download from S3 |
| v2 — cpuopt1 | ECR acesense-gpu:bench-5ba30d6-cpuopt1 | (1) FFmpeg/NVDEC path wired in read_video_optimized (2) parallel ball HoughCircles (3) MediaPipe workers cap 4→~12 |
All SM numbers below say which version. Lambda Labs numbers from earlier July are out of scope for this SM-first plan (see MasterContext).
2. Approved resources (usable now)
2a. SageMaker — us-east-1 only for real GPU training
Training job usage = 1 (unless noted):
| Family | Approved training types (1-GPU unless noted) |
|---|---|
| g6 (L4) | ml.g6.xlarge … 16xlarge (also 12xlarge = 4×L4) |
| g5 (A10G) | ml.g5.xlarge … 16xlarge (also 12xlarge = 4×A10G) |
| g4dn (T4) | ml.g4dn.4xlarge, ml.g4dn.8xlarge only |
| p4d | ml.p4d.24xlarge = 8×A100 (training=1) |
| trn1 | ml.trn1.2xlarge (not CUDA AceSense) |
Also: endpoint quotas for many g4dn/g5/g6 including *.4xlarge (=1); processing all ml.g6.* =2.
AceSense RAM floor ~64 GB → practical training SKUs: *.4xlarge and up. Skip xlarge/2xlarge for this pipeline.
Other regions (eu-*, us-west-2, …): no useful GPU training quota (endpoint leftovers only at 16–32 GB).
2b. EC2
| Region | G/VT vCPU | P vCPU | Practical max GPU host |
|---|---|---|---|
| us-east-1 | 8 | 0 | g4dn/g5/g6.2xlarge (8 vCPU / 32 GB) |
| eu-central-1, eu-north-1 | 8 | 0 | same |
| others checked | 0 | 0 | — |
Inf/Trn On-Demand 8 in us-east-1 — irrelevant for CUDA AceSense.
2c. Vertex (GCP) — previously used
Custom Job GPUs were available enough to complete L4/T4/V100 benches (2026-07-12). Re-auth required for new submits as of 2026-07-13 evening.
3. Benchmark results
Wall = pipeline WALL_SEC inside container (not full cloud billable time). Cold start (provision + pull) is extra ~3–6 min typical on SM/Vertex.
3a. SageMaker — version v1 (fix1-sm)
| RUN_ID | Label | Instance | Wall (s) | Court | Ball | Players | Pose (s / fps) | Enrich |
|---|---|---|---|---|---|---|---|---|
20260713_020318 | l4 | ml.g6.4xlarge | 486 | 52 | 136 | 60 | 156 / 30 | 24 |
20260713_020318 | t4 | ml.g4dn.4xlarge | 871 | 75 | 228 | 116 | 353 / 13 | 20 |
20260713_022702_highcpu | l4-hc | ml.g6.8xlarge | 489 | 51 | 143 | 60 | 155 / 30 | 24 |
20260713_022702_highcpu | t4-hc | ml.g4dn.8xlarge | 872 | 67 | 235 | 118 | 361 / 13 | 20 |
20260713_143430-maxcat | l4-max | ml.g6.16xlarge | 505 | 47 | 161 | 60 | 156 / 30 | 24 |
20260713_143430-maxcat | t4-max | ml.g4dn.8xlarge | 856 | 67 | 232 | 116 | 350 / 13 | 20 |
20260713_143430-maxcat | a10g-max | ml.g5.16xlarge | 443 | 55 | 135 | 48 | 111 / 42 | 27 |
20260713_143430-maxcat | a100-max | ml.p4d.24xlarge | 331 | 46 | 119 | 32 | 54 / 87 | 18 |
Findings (v1): Extra CPU does not help. A10G beats L4. A100 fastest SM but only 1 GPU used of 8. T4 too slow for prod target.
3b. SageMaker — version v2 (cpuopt1)
| RUN_ID | Label | Instance | Wall (s) | Court | Ball | Players | Pose | Enrich | Video |
|---|---|---|---|---|---|---|---|---|---|
20260713_154338-cpuopt | a10g-cpuopt | ml.g5.4xlarge | 430 | 63 | 96 | 48 | 111 / 42 | 12 | 41 |
20260713_154338-cpuopt | l4-cpuopt | ml.g6.4xlarge | 494 | 52 | 133 | 60 | 157 / 30 | 11 | 35 |
vs v1: A10G wall 443→430; ball 135→96; enrich 27→12. L4 ≈ flat (486→494). Video load time rose on v2 (NVDEC path / accounting) — net still small A10G win. g5.4xlarge ≈ g5.16xlarge for this workload.
3c. EC2 — version v1, max under G=8 (*.2xlarge, 32 GB + swap)
RUN_ID: 20260713_144007-ec2-maxcat
Do not read the wall times as “L4/A10G are slow on EC2.” Core GPU stages were in a normal-ish range; walls exploded because player detection thrashed on 32 GB RAM (full-res frames + swap).
| Label | Instance | Wall | Ball | Pose | Players | What happened |
|---|---|---|---|---|---|---|
| ec2-l4-max | g6.2xlarge | 2159 | 204 | 255 | 1285 | Players ~20× SM; swap cliff |
| ec2-a10g-max | g5.2xlarge | 1887 | 161 | 190 | 1140 | Same pattern |
| ec2-t4-max | g4dn.2xlarge | 949 | 226 | 364 | 120 | Players stayed normal; wall still ~2× SM |
For comparison, SM A10G players ≈ 48s, ball ≈ 96–135s, wall ≈ 430s.
Finding: EC2 at current G/VT=8 quota is not a viable AceSense host — not because the GPU can’t run TrackNet/pose, but because 32 GB is below the ~64 GB floor and player stage falls off a cliff. Use SM numbers for planning, not these EC2 walls.
3d. Vertex — version v0 (2026-07-12)
Image: us-central1-docker.pkg.dev/acesense-prod/acesense-gpu/pipeline:bench-5ba30d6-fix1
RUN example: gs://acesense-prod-gpu-bench/vertex-bench/runs/20260712_162418/
| GPU | Machine | Wall (s) approx |
|---|---|---|
| L4 | g2-standard-16 | ~569 |
| T4 | n1-highmem-8 | ~1079 |
| V100 | n1-highmem-8 | ~580 |
SM L4 (~486) beat Vertex L4 (~569) on same era code. Vertex v2 cpuopt not run (gcloud auth expired).
4. Recommended SKUs (from data)
| Use | SKU | Why |
|---|---|---|
| Default SM inference | ml.g5.4xlarge | Best 1-GPU price/perf in quota; ~430s v2 |
| Fastest SM now | ml.p4d.24xlarge | ~331s; wasteful until multi-GPU code |
| Avoid for prod target | T4, EC2 *.2xlarge, SM *.xlarge/2xlarge | Slow or OOM |
5. Artifacts
- SM results:
s3://acesense-gpu-bench/sagemaker-bench/runs/<RUN_ID>/<label>/timings.json - Vertex results:
gs://acesense-prod-gpu-bench/vertex-bench/runs/<RUN_ID>/ - Scripts:
acesense-gpu-backend/deploy/{sagemaker,vertex,ec2}/